This case is about a project of an online car financing simulator, as a bank product which provides users an estimative of values for bank fundings for the purchase of vehicles according to information provided about the vehicle and the client.
In this context, it will be presented a story in chapters:
1. The old version
2. The problem
3. The hypothesis
4. The validation process
5. The A/B testing - part 1
6. The A/B testing - part 2
7. Considerations
8. The takeaway
Final product snapshots mobile with Black and white filter to use as cover
1. The old version
The product and the context
The car financing simulator is a responsive web application available at a bank website, and the main goal of the product is to be the bank platform to get leads as the first step on the conversion funnel of leads into customers.
The product consists of an application which calculates an estimative of the bank financing values available per user to purchase a car based on users credit rating, where the user start by seeing an approximate amount to be paid in instalments for a financing contract, and then finish the cycle by going to a parter car shop to check final prices with the bank, send a formal request through the car seller and get the final approval from the bank to close the deal and get the money to buy the car.
To simulate the value to be paid, the user must follow the flow which starts with a form to enter information about the vehicle and the the purchaser, and the last screen where the user can manipulate and check the amount and value of instalments according to the down payment (he can can also change this value and see the alterations on the instalments as well).
First version of the page
To get into the simulator the user usually starts his journey searching online for bank fundings for cars purchasing and getting as results the links to the simulators of the biggest banks in Brazil.
Then the user goes to each bank page and simulates to check the best conditions, and after getting to know the estimative of how much he would pay on each bank, he goes to a car shop where the seller will check the real conditions for the funding in each bank and then send the funding request to the chosen one (chosen by the user).
Considering this simplified journey, the scope of this product is from the online simulation on the bank website to the moment the user goes to the car shop (more likely to choose closing the deal with the bank).
User flow from search on google to the last screen of the simulator flow
Simplified user journey - Simulator flow
2. The problem
Data analysis and research - findings
Considering the simulator scenario, we had a flow starting with the simulations and ending with the user’s funding requests being approved by the bank and these users being converted into bank’s clients with contracts.
Data snapshots
The simulator's scope includes the amount of user's financing requests before the final approval (this step is under another team/product scope), and through data analysis we could find a big gap between the simulations and financing requests.
There was also quantitative research with users on the simulator page with questions and also a qualitative research through interviews with recent clients who had used the simulator before contracting the financing with the bank, and among the discoveries there were some interesting and relevant informations:
3. The hypothesis
The suggestion for a new version
After collecting all information about the simulator performance, users and business pain points, the team put the pieces together to come up with a new version with new features:
4. The validation process
Testing, iterating and testing again
Once we had the first new version proposal, the next step was a validation process starting with a face-to-face pilot test to check the main usability issues before running an online usability test with a selected group of people.
The most relevant discoveries from the pilot test were:
Snapshots of chatbot step and results page
Once we hadn’t negative feedback about the chatbot step we decided to keep it in future iterated versions to be tested, and after the findings from the test and considerations, 2 other versions were developed to improve the user experience considering the results (and to be tested with 2 different groups of people):
1 - One improving what was noticed in the results page tested, with the results in a single screen
2 - A second one as another hypothesis with the results in a flow with steps divided into different screens
The chatbot step was kept.
The flow of the version with results with more screens had 4 steps:
1. Choosing the down payment and instalments
2. Insurance options
3. Search for the nearest bank partner car shops
4. Results with summary of simulation, offers from the bank portal of car sales and the results list with the nearest partner car shops
Each step had its own screen and space minimising the cognitive efforts from the user to identify and interact with each element.
- A fixed bottom bar was created to keep visible the summary information about the car and the total price of the financing contract
- The flow was created to guide the user through all items which he must choose and then show him the results in the summary with the items included and its respective price
- It was available the option of sending the summary via e-mail instead of the option of downloading the summary in the device
The test and the results
The 2 versions were tested with 2 different groups of people who fit the behavioral requirement of having the intention of contracting a car financing in a near future, and also other demographics characteristics from the customer data base.
The test was runned on an online platform called Testaisso focused on remote usabily test and the most relevant findings were:
Results from usability test
Conclusions and what to do next
After gathering the results we realised that the test wasn’t very conclusive as they were not very different in general and also considering the feedback from testers regarding results page informations and features, therefore we made some decisions:
5. The A/B testing part 1
The chatbot format vs. the form
The first A/B testing was runned between January 2021 and April 2021 with 10% of visitors traffic for each version and these are the results:
Final variants for the 1st part of the A/B test
6. The A/B testing part 2
The new version of results and the old one
The second A/B testing started on May 5th 2021, but we only have accurate data from the period between May 12th and June 28th due instability of the analytics tool.
The test confronted the variant A as the old version of results where it was displayed just the down payment and the instalment options, while the new proposal (variant B) had some new features and Adobe Analytics allowed us to track interactions with 2 of them: the clicks on the saving summary option, and the clicks on offers from the bank portal of car sales.
Final variants for the 2nd part of the A/B test
Therefore we can see the new version is getting better results, which means it was decided that the simulator would keep the first step of the flow where users input informations about themselves and the car they want to purchase as a form to fill the required fields, as it had better results on the first AB test compared to the chatbot version, and the results page would be the new version proposed as we could check that the new one got better results in the second AB test compared with the old page where it was provided as result just the instalments and the car total price.
Live site link (the final project is the second part of the flow after providing information required in the form).
More pictures of the new version tested:
Final design - 1st fold of results page
Final design - summary of simulation to download and save
Final design - Carousel of insurances
Final design - Carousel of car ads from MeuCarroNovo platform - results according to inputs provided during simulation
Final design - Section to search for partners car shops
7. Considerations
And other details
Why is this not a conclusion yet? Well, the whole story might seem to had followed a flow but actually there was some details in the middle which had influence in the final version used in the A/B testing.
Along the process we had the opportunity to run a moderated usability testing (a whole another case) and interviews with a group of recent clients to get more assertive insights and informations about user’s experience with both hypothesis (results in a single screen and results in a steps flow) and then we could implement the more feasible iterations before running the A/B testing, and after the usability test we organized all the discoveries, presented them to the team and stakeholders along with a MoSCoW based workshop to get new ideas, organize and evaluate them considering efforts and blocks, to create a backlog of features and improvements for the product.
.
8. The take away
The learnings from the case
Despite the process did not follow a perfect planned path, considering the A/B testing results until now I believe we are going in the right direction.
As usually happens, in this design process we faced problems such as data analytic tools with constrained configurations, ineffective tagging from long time ago and long waiting time for development due the amount of bugs and issues, however we could follow a process and take decisions based on tests and research, which is now getting us into the data driven action mode in the right way. We are starting to get real and accurate data to analyse and therefore support business decisions.
Last but not the least, it was also possible to show the stakeholders that having a design vision at the core of the projects can bring tangible results and this was possible because of the people working in this project: the Product Designer Ana Pegowho had started since user journey and came up with the first version tested by me in the pilot test at the point when I took over, and the UX Writer Glaucia Lopes who has been always helping us to create a better user experience.
