Augmented Reality Banking with TypingDNA
We dream about a world where displays will be much more than squares that sit in our hands. This future is coming closer with the power of Augmented Reality (AR). In an utopic future where people see more with their eyes through technology we imagine that contextual information should be available without looking down at screens anymore.
We provide an experience where the user can see the banking information such as total balance, income and outcome just by pointing the camera at the card.
To protect the privacy of our users, we have incorporated a secure way of authentication and all the processing is done locally on the device.
We used RealityKit from Apple to create a dynamically generated experience in AR to provide banking data such as total balance, income and outcome.
To register and authenticate our users we used TypingDNA API with the "same-text" type which is evaluated on the last 8 digits of the card. This was possible by integrating the TypingDNA Recorder for iOS.
We made use of Card.io library to recognise the card numbers and take a cropped picture of the card to use it as a tracker in the AR experience.
5 out of the 6 members were new to programming. We spent 2/3 of the time on planning and studying programming together. Because of our limited resources, we wanted to build something on a common platform making use of JavaScript and other web-based technologies.
We wanted to create an interactive experience where the user would be able to use their hands to be recognised through the TypingDNA by extending the recorder to understand the hand pose estimation data. However, we encountered major performance difficulties and we were not able to pursue this further on the web.
After we realised that our dream cannot be achieved without a native technology we were very demotivated after spending 40+ hours studying web technologies.
We decided to make our dream a reality through iOS, but only 1 out of 6 had access to an iPhone and a MacBook (remember the professional developer? haha 🥲). But we did not lose hope and we pair-programmed together using TeamViewer, coded logic in Swift Playgrounds online but most importantly, we created virtual machines and spent hours getting Xcode in a virtual environment up and runnning.
Finally we were only left with 2 weeks before the deadline and we ACTUALLY got to start developing.
Not to say.. Apple did not approve our application on TestFlight so we decided to go on our own and use AppDistribution by Firebase which gives us the flexibility we need for this project.
Since we all have our jobs and studies it was incredibily difficult to schedule times to code together particularly during the week. This has also been difficult because of the differences in the timezones.
- Everyone in the team knows now how to code
- Most members are now proficient in JavaScript and TypeScript after doing all the exercises on freeCodeCamp (even though we didn't end up using it for the final app)
- Most of us did a crash course on SwiftUI
- Finalised a demo where you can scan a card, get the card numbers, register and authenticate through the API to enjoy a demo AR Experience
Web is too weak for our dreams. Oh and JavaScript, TypeScript, Swift and TypingDNA API.
Firstly, we would love to integrate hand pose estimation in Swift and adapt the recorder to register a typing pattern without using the keyboard.
Secondly, we would like to connect through OpenBanking to the bank account and provide real live data information in the AR experience.
Finally we would like more than a MacBook and an iPhone to be able to test our dream...