Breakout Session – how to prototype your enterprise project hackathon-like

This is the story of my team creating something awesome within one day. It begins in November of 2017 at “Hack your Office”, a 24-hour hackathon hosted in cooperation by my employer synyx and our customer dm-drogerie markt. Although it was an excellent hackathon, this is not the day I am refering to but it was on this day when the idea was born. Several of my team members from dm where participating in the hackathon, even Matthäus – one of our product owners – joined us. He didn’t contribute anything to the code but he was absorbed in the electrifying atmosphere of everybody being excited to hack something together. The amazing thing about hackathons is that everybody has the intrinsic motivation to be creative and to work hard to produce something awesome in the short time that is available, while having fun! Matthäus formed the vision to experience something similar with our own, regular development team from the dm office.

The Vision

The vision was plain and simple: Create a working POC-like product that adds real value and runs in production within a single day while working in a hackathon-like atmosphere.

On this day we would not work in our usual office but together in a remote location, focussing as a team on this one goal alone. We called it “breakout session”. Luckily we had a suitable use case in the pipeline. The digital receipt system of dm (“e-Bon”), that was reaching its end-of-life in 2018 for several reasons, had to be reimplemented. We defined the ambitious goal that within a day we should be able to walk downstairs to the real dm store with our phone, actually buy something and see the genuine receipt created by the check-out counter displayed in our dm customer account on our phone. Everything with production systems and production data.

Preparation for BonBon

Although it was intended as a one-day thing we didn’t want to go in blindly. Some preparation was necessary to reduce the risk of failure in our one-day adventure.

First we made up a small concept about basic things like where the data had to come from, what system our product will be running on, which systems would have to talk to each other. The actual business use case was worked out in more detail by our product owners.

We knew we would be developing a new microservice so we needed a system to run it on. At dm there is a provisioning platform in place that can pull up a fully configured virtual machine cluster on multiple stages including firewall rules, load balancing and everything within half an hour. However we did this beforehand because experience tells us that this really cool technology doesn’t always run smoothly on first try due to its massive complexity.

The receipt data from the ~9000 checkout counters of all dm stores in europe is stored in real time on some central backend system so we prepared a route piping the data in real time into a topic on one of our Apache Kafka clusters. We limited the data to the last two days, which should be more than enough for our POC.

We assembled the team for our breakout session having cross-functionality in mind as we had to master tasks in backend development, data processing, operations and mobile development on that day. Our regular development team (usually in charge of the customer backend of dm) formed the backbone of the breakout team. We already incorporate backend expertise and extensive knowledge in operations and many areas of data processing thanks to our devops culture. Unfortunately we are a bit thin on frontend and mobile development so we asked colleagues from the mobile team to join us. One guy from analytics helped us setting up Kafka and the server operations team was (as always) on stand by in the office and informed about our endeavour in case anything complicated would go wrong with the infrastructure.

Finally we needed a location to retreat to. One of my teammates arranged three rooms at the office of his employer diva-e Netpioneer, which is beautifully located at the city park in Karlsruhe – quickly accessible by bike, train or car for all participants. He also made reservations for lunch at the nearby pizza place – one more thing of importance that we would not have to bother about on the breakout day.

The discussion about the upcoming solution’s name was held some day during lunch. We called it BonBon.

The Session

Finally the day arrived on a Wednesday in March. Everbody was hyped about it, though some of us a bit more sceptically hyped. Ten developers, two product owners, two interested guests from the business department and one scrum master were gathering in the kitchen at diva-e, having the first of countless coffees and enjoying breakfast pretzels eager to abuse our keyboards for some dirty hacking. Our product owners started the session at 9:00 with a short kick-off introducing the details of the use case and we quickly forged the requirements into small user stories that would eventually add up to the desired POC.

Twenty minutes in the user stories were ready and the signal to commence the hacking was given. Suddenly the excitement unloaded in a stream of productivity! Small teams were vibrantly working to put the pieces together. Creating repositories, initializing a Spring Boot app, testing out infrastructure all around within the first few minutes! Thanks to modern development tools the results came in quickly. Within one hour the first API was useable in a freshly created Backend Service. Shortly after the new born iPhone app was capable to authenticate users at the checkout counter via QR-code. Before lunch the Kafka data trickeled in at the backend. A rudimentary build and release process was in place. This didn’t feel like work, it was the hackathon atmosphere transferred to our everyday activity.

The mode we were working in was mainly in small teams of 2-3 developers tackling the different tasks in pair programming. Every hour we did a quick stand up to sync our progress. Small successes were celebrated, problems announced to find a fast solution using the creative power of the whole team. The product owners where always within reach, researching business details, answering arising questions and giving feedback on the progress. Our scrum master was scurrying around the team during development enabling communication, removing impediments, moderating stand-ups, giving impulses.

Unfortunately the lunch break did not work out that well. Visiting a pizza place with a group of 15 unexpectedly turned out to take a lot longer than anticipated. A huge chunk of our precious time and focus was lost to some delicious slices of Italian culinary art.

During the time between lunch and the contemplated end at 17:00 we managed to develop almost everything that was necessary to complete our goal. There was one problem left with processing the huge amount of real time data and one connectivity problem between the app and the prod backend. Some team members committed to two more hours of problem solving which payed off big time at the end of the day!

Around 19:00 a colleague went downstairs to the local dm store and got some apple-cinnamon cereal. He went to the counter and held his phone to the QR Code scanner to let the cash identify his account. After payment he opened the BonBon app and the receipt of the purchase was displayed with article description, product pictures, prices and everything. Achievement unlocked!

Unfortunately we missed the opportunity to take a picture of this moment but the receipt in the final app prototype looked like this:

Hacking around problems

To enable this quick success we had to make our hands a little dirty by cutting corners on otherwise indispensable practices of our everyday work. (Except security, of course. Access to the APIs and the data was secured the whole time.) The whole day we were running in full-on prototype-POC mode. We agreed in advance that all produced code had to work quickly and didn’t have to be beautiful or readable at all. Tests were omitted from the beginning. Copy-paste from other projects or Stackoverflow for quick results was encouraged. Upcoming obstacles were circumnavigated with the quickest workaround instead of looking for the “correct” solution:

• Connectivity from the app to the BonBon production system was created last minute with some obscure temporary routing through the release gateway into the DMZ.
• Deployment was done by copying the jar artifact to the production vm “on foot”.
• The receipt data was stored in memory – without backup. We just left out the persistence layer to save time.
• The huge amount of data accumulated in 2 days hit us unexpectedly. We just shrugged it off and hoped the system performance would endure the load.
• Debugging problems by frantically adding new logs on the new production system was common practice
• … and some more