Breakout Session - how to prototype your enterprise project hackathon-like
This is the story of my team creating something awesome within one day. It begins in November of 2017 at "Hack your Office", a 24-hour hackathon hosted in cooperation by my employer synyx and our customer dm-drogerie markt. Although it was an excellent hackathon, this is not the day I am refering to but it was on this day when the idea was born. Several of my team members from dm where participating in the hackathon, even Matthäus - one of our product owners - joined us. He didn't contribute anything to the code but he was absorbed in the electrifying atmosphere of everybody being excited to hack something together. The amazing thing about hackathons is that everybody has the intrinsic motivation to be creative and to work hard to produce something awesome in the short time that is available, while having fun! Matthäus formed the vision to experience something similar with our own, regular development team from the dm office.
The vision was plain and simple: Create a working POC-like product that adds real value and runs in production within a single day while working in a hackathon-like atmosphere.
On this day we would not work in our usual office but together in a remote location, focussing as a team on this one goal alone. We called it "breakout session". Luckily we had a suitable use case in the pipeline. The digital receipt system of dm ("e-Bon"), that was reaching its end-of-life in 2018 for several reasons, had to be reimplemented. We defined the ambitious goal that within a day we should be able to walk downstairs to the real dm store with our phone, actually buy something and see the genuine receipt created by the check-out counter displayed in our dm customer account on our phone. Everything with production systems and production data.
Preparation for BonBon
Although it was intended as a one-day thing we didn't want to go in blindly. Some preparation was necessary to reduce the risk of failure in our one-day adventure.
First we made up a small concept about basic things like where the data had to come from, what system our product will be running on, which systems would have to talk to each other. The actual business use case was worked out in more detail by our product owners.
We knew we would be developing a new microservice so we needed a system to run it on. At dm there is a provisioning platform in place that can pull up a fully configured virtual machine cluster on multiple stages including firewall rules, load balancing and everything within half an hour. However we did this beforehand because experience tells us that this really cool technology doesn't always run smoothly on first try due to its massive complexity.
The receipt data from the ~9000 checkout counters of all dm stores in europe is stored in real time on some central backend system so we prepared a route piping the data in real time into a topic on one of our Apache Kafka clusters. We limited the data to the last two days, which should be more than enough for our POC.
We assembled the team for our breakout session having cross-functionality in mind as we had to master tasks in backend development, data processing, operations and mobile development on that day. Our regular development team (usually in charge of the customer backend of dm) formed the backbone of the breakout team. We already incorporate backend expertise and extensive knowledge in operations and many areas of data processing thanks to our devops culture. Unfortunately we are a bit thin on frontend and mobile development so we asked colleagues from the mobile team to join us. One guy from analytics helped us setting up Kafka and the server operations team was (as always) on stand by in the office and informed about our endeavour in case anything complicated would go wrong with the infrastructure.
Finally we needed a location to retreat to. One of my teammates arranged three rooms at the office of his employer diva-e Netpioneer, which is beautifully located at the city park in Karlsruhe - quickly accessible by bike, train or car for all participants. He also made reservations for lunch at the nearby pizza place - one more thing of importance that we would not have to bother about on the breakout day.
The discussion about the upcoming solution's name was held some day during lunch. We called it BonBon.
Finally the day arrived on a Wednesday in March. Everbody was hyped about it, though some of us a bit more sceptically hyped. Ten developers, two product owners, two interested guests from the business department and one scrum master were gathering in the kitchen at diva-e, having the first of countless coffees and enjoying breakfast pretzels eager to abuse our keyboards for some dirty hacking. Our product owners started the session at 9:00 with a short kick-off introducing the details of the use case and we quickly forged the requirements into small user stories that would eventually add up to the desired POC.
Twenty minutes in the user stories were ready and the signal to commence the hacking was given. Suddenly the excitement unloaded in a stream of productivity! Small teams were vibrantly working to put the pieces together. Creating repositories, initializing a Spring Boot app, testing out infrastructure all around within the first few minutes! Thanks to modern development tools the results came in quickly. Within one hour the first API was useable in a freshly created Backend Service. Shortly after the new born iPhone app was capable to authenticate users at the checkout counter via QR-code. Before lunch the Kafka data trickeled in at the backend. A rudimentary build and release process was in place. This didn't feel like work, it was the hackathon atmosphere transferred to our everyday activity.
The mode we were working in was mainly in small teams of 2-3 developers tackling the different tasks in pair programming. Every hour we did a quick stand up to sync our progress. Small successes were celebrated, problems announced to find a fast solution using the creative power of the whole team. The product owners where always within reach, researching business details, answering arising questions and giving feedback on the progress. Our scrum master was scurrying around the team during development enabling communication, removing impediments, moderating stand-ups, giving impulses.
Unfortunately the lunch break did not work out that well. Visiting a pizza place with a group of 15 unexpectedly turned out to take a lot longer than anticipated. A huge chunk of our precious time and focus was lost to some delicious slices of Italian culinary art.
During the time between lunch and the contemplated end at 17:00 we managed to develop almost everything that was necessary to complete our goal. There was one problem left with processing the huge amount of real time data and one connectivity problem between the app and the prod backend. Some team members committed to two more hours of problem solving which payed off big time at the end of the day!
Around 19:00 a colleague went downstairs to the local dm store and got some apple-cinnamon cereal. He went to the counter and held his phone to the QR Code scanner to let the cash identify his account. After payment he opened the BonBon app and the receipt of the purchase was displayed with article description, product pictures, prices and everything. Achievement unlocked!
Unfortunately we missed the opportunity to take a picture of this moment but the receipt in the final app prototype looked like this:
Hacking around problems
To enable this quick success we had to make our hands a little dirty by cutting corners on otherwise indispensable practices of our everyday work. (Except security, of course. Access to the APIs and the data was secured the whole time.) The whole day we were running in full-on prototype-POC mode. We agreed in advance that all produced code had to work quickly and didn't have to be beautiful or readable at all. Tests were omitted from the beginning. Copy-paste from other projects or Stackoverflow for quick results was encouraged. Upcoming obstacles were circumnavigated with the quickest workaround instead of looking for the "correct" solution:
- Connectivity from the app to the BonBon production system was created last minute with some obscure temporary routing through the release gateway into the DMZ.
- Deployment was done by copying the jar artifact to the production vm “on foot”.
- The receipt data was stored in memory - without backup. We just left out the persistence layer to save time.
- The huge amount of data accumulated in 2 days hit us unexpectedly. We just shrugged it off and hoped the system performance would endure the load.
- Debugging problems by frantically adding new logs on the new production system was common practice
- … and some more
Operating this way we ended up with a working proof of concept covering everything we aimed for on this day, which was a huge success! But to be clear: The thing we produced is obviously far from being actually production ready. Some parts of the code need refactoring to meet our quality standards. Test coverage has to be provided everywhere on all layers of the test pyramid. Data has to be persisted and resilience measures have to be taken to achieve acceptable robustness of the solution. A proper deployment process has to be established, maybe with containerization. Continous integration and continous delivery have to be provided. The app has to be refurbished, approved and published. And so on.
Some aspects of the electronic receipt use case were not possible to implement on the breakout day - like the original idea to avoid paper waste for the receipt. To stop the checkout counter from printing the receipt we would have to change and deploy the software running on it, which is maintained by another team, rolled out countrywide and is not integrated into a CD lifecycle - impossible to achieve in one day.
Ultimately the day turned out to be exceptionally valuable both as team building excercise and as an effective method of starting a project. We proved our idea to be feasable in a real production example. We have a working code base, working artifacts and infrastructure, that can be built upon and expanded. This will be pursued in the form of normal backlog items during our usual sprint-to-sprint routine. The team gained a decent boost of motivation and team spirit - not only the developers but also the product owners, who worked in support of the team and the use case the whole day, and our scrum master, who enabled focus and kept our spirits high with his subtle (*cough*) positive nature.
We perceived the breakout session experiment as a successful, wholesome experience and can recommend it as method for POCs, project starts, feature kick-offs to every team that wants to try something new.