Maturing your Route to Go-Live for Citizen Analysts

Are you a data engineer, data scientist or data analyst working in a business team? 

Within large and small organisations, having teams with technical data skills sit outside technology or engineering teams is becoming a growing trend.  For example, data analysts in finance teams who are comfortable with Excel and SQL skills are now using advanced tools like Databricks and PowerBI. 

With access to these new technologies, business users can quickly drive new value into their organisation by automating and scheduling processes and deriving new insights to maximise revenue, save costs or improve efficiency.  As usage grows, a pattern we often see is that the number of artefacts to manage becomes larger and larger.  Structured organisation of code and version control becomes important.  The impact of errors grows, and users begin to need the ability to test their changes and releases in a robust manner. 

These challenges are all solved (well solvable) problems in the realm of data engineering and DevOps, but adopting, developing and maintaining a mature software development lifecycle is generally not something that business teams often consider. 

If most of the statements below ring true, you may need to think about reviewing and improving your route to go-live. 

 My team only really use the production environment and data for development and testing. 

• If we need a test data set, I can just create a table with a copy of production data in it and use that. 

• If we make a mistake, we could accidently break production processes that are relied upon by the business. 

• We need to be careful when developing because we could interfere with production processes. 

• We don’t have documented release/rollback processes. 

• My codebase is growing in size and complexity over time. 

• We don’t keep a history of versions of our reports/code.  If a key piece of code is deleted, we would need to re-write it. 

• All testing that we do is manual, written for each change and designed and executed by the user that made the change. 

If you are ready to review your route to go-live, ask yourself the following questions:

Where is your work stored? 

We frequently see analysts storing work on their local machines or in departmental shared drives.  Some tools like Power BI and Databricks provide “user areas” as a part of the service and while these areas offer flexibility, you will need to understand how they are used and who has access.  For example, if an important piece of work-in-progress is stored in a user area and that user cannot work for a few days, then it could be very hard for a colleague to pick up and progress the solution.  Integration with a centralised version control provider may be an option, however there are complexities to consider such as security and access, branching strategies and user training. 

Development, Validation and Testing 

Is it possible and easy for users to test their work before using it to make business impacting decisions?  It is often possible to create isolated environments for development or test purposes, but these will take time to create and maintain.  If isolated environments are the chosen solution, then making sure that the appropriate data is available and maintained can add an overhead.  Depending on the business impact your change can have, should it go through any automated or manual review or approval processes before being used? 

Release/Deployment processes 

Once development work has been completed and tested, how do you make the change in the production environment? 

This may be simple and manually done if you need to release a small change to a single report/notebook, but what if the deployment involves 10s of individual artefacts that need to be released in a specified order.  If the deployment fails for some reason, is it easy to recover or rollback to a previous version and how long would this take? 

Post Release Testing 

Once a piece of work has been released, how do you verify that the change is behaving as expected? 

Are the results correct? Is the performance acceptable? Are the costs expected? 

What would you do if you find a problem one minute, one day, one week after release.   

Here are some examples of how a basic “route to go-live” (yellow) can be matured (blue) as production processes become more critical.  Further improvements beyond this can also be made if required. 

Summary 

Tools and technologies that were historically used by developers and engineers are now increasingly being used in the business.  Therefore, it is important that some of the lessons learned over decades of software development practice start making their way into new areas. 

The most appropriate “route to go-live” for your team can range from a very simple manual process all the way up to a fully automated continuous delivery process that introduces new tools to purchase, manage and maintain. 

Start with gaining an understanding of how changes are made within your team and map out your existing route to go-live process. 

 Investigate the capabilities included in the tools that you currently use.  For example, Power BI and Databricks both offer git integration.  Power BI has the concept of deployment pipelines.  Azure DevOps and GitHub both offer source control as well as ways to automate testing, packaging and deployment of your solutions.  If you need more flexibility than the native integrations allow, then DevOps Releases and Pipelines or GitHub Actions may provide the capabilities needed at the cost of extra complexity. 

Once you have an idea of what is possible, you can identify the areas for improvement in your “route to go-live” that are terrifyingly risky operationally, worryingly slow, costly or that will be unmanageable as your team grows. 

Map out a high-level idealised process, and then based on priorities aligned to the areas for improvement, you can plan the work to improve your processes.  With a good view of where you want to get to, these changes don’t need to be made in one go.  If software development and DevOps processes are not a core part of your job role, then learning and embedding good practices over a longer period may be the best approach. 

Adding automation by learning new skills and implementing very robust DevOps processes may sound exciting but also may be overkill for your scenario so ensure that you assess the value of making the improvements. 

If you need any help with this or anything data related, please contact us on info@arreoblue.com 

About Arreoblue   

Arreoblue is a forward-thinking solutions provider specialising in tailor-made strategies to optimise business processes and foster growth. With our Assess, Accelerate and Amplify methodology, our experts will utilise their decades of experience to empower your people with a platform of success and solution that works FOR you.    

To find out more, get in touch with one of our dedicated team today at info@arreoblue.com