Critic of Agile SDL
This article aims at criticizing the Microsoft Agile Secure Development LifeCyle methodology. And by criticize I do not mean despise but objectively judge and give my opinion about what I found useful and what I found unrealizable or enhanceable. My objective here is to help developers to integrated security more efficiently in their development and ultimately propose a methodology merging Agile practices and Security principles. This methodoly is an adaptation of the traditionnal Microsoft SDL:
First you should be familiar with Agile methodologies, the most known are Kanban and Scrum. Today’s article is going to focus on Scrum as an example as it’s the one I practiced most and most familiar with. You may also give a look at Microsoft website for Agile SDL.
First reproach, the website is damn poor… An interactive table with a 2 lines descriptions for each point. No examples, no advice, no picture. I would have expected a little more help on such interesting and important topic. I do believe that integrating security in agile development process is an important,passionating and critical mission, and Microsoft do have the means and the brilliant engineers and project leaders to provide a much wider and deeper experience and insight on that matter. I was really disappointed to see the lack of resources or help on this subject even from the creators of the method…
From my point of view it doesn’t fit most of companies. The way Microsoft describes it would need a security expert as part of any Scrum team and let’s face it most companies can’t afford it. I was educated and doing my career in middle size software companies with teams from 2 to 8 persons in agile practices. I NEVER encountered a security dedicated engineer im my team… Most of the time we would use security consultant for pentesting or security reviews at best. Later on, when I became the security consultant I was facing the same problem. I was the security guys advicing developping teams that had no clue about security most of the time. Of course there was exception but as a developer it’s really hard to keep the pace on knowing all new security attacks / trends as well as mastering languages and new technologies. All that to say that my opinion on Agile SDL tends to adapt it for not-so-big companies and teams.Then let’s start looking at the content of the methodology.
The Methodology itself
It is divised in three sub-part which names are pretty straight forward :
- Every Sprint Practices : Security practices you should perform during each sprint. Therefore they should be added to your Definition of Done.
- Bucket Practices: This one is much more blur… Microsoft says you should do it once in a while… not every sprint but not only one time either… My personal opinion would be to do it prior to any major release (usually 4-6 sprints).
- One-Time Practices: These should be done one time. Most of them involves requirements prior to development or preparation “in case the worst happen”. Once settled you should refer to it for ongoing development. On this part I strongly disagree… Agile process is about iterating, doing requirements and stick to it while developing is against my vision of Agile. I therefore recommend to refresh these requirements every time you can because in Agile one-time practices should not exists, Agile should be about adapting and moving.
Then we can dive into each sub-part with the recommended practices:
Every Sprint Practices:
- Core Security Training: it states that every member of the team should at least attend a security training once a year. Both to discover security aspect or to keep up to date about security matters. I agree on this one because it is really necessary to make developers aware about the security aspect and consequences of their job. However I have to admit I don’t know why they put it in the every sprint practice… It should be added to the training plan of the team for the year and therefore I’d better see it in the one-time practices…
- Use Threat modeling : Apply a structured approach to threat scenarios in order to map risk and security vulnerabilities. Well this is maybe the point where most of people will disagree with me but I don’t believe in Threat Modeling to identify vulnerabilities. Most of the time it’s an after-job, addind lot of documentation and tremendous effort for a arguable gain… Therefore I wouldn’t recommend to implement this and especially not for every sprint.
- Use Approved Tools : This is about education and good practices associated with the tools you are using. At the end this practice is about setting a wiki to share security good practices about : compiler options, setting up virtualhost or virtual machine, secure deployment, build processes. Once again I do not believe it belongs to this category but you should make sure you have a section on your wiki about secure best practices for the tool you are currently using.
- Deprecate unsafe functions: This section is clearly about the code. Deprecated and especially unsafe functions should be replaced. This belongs here but can be easily automated using tools such as SonarQube, Jenkins and LINT or PSR rules and analysis. Just be sure to effectively replace the deprecated functions
- Perform Static Analysis: Code analysis can be performed using tools depending on your language. Basically this is just a huge text search looking for unsafe code that would lead to SQL Injection or other problems. The functions are not necessarily deprecated but dangerous (best examples are exec and eval commands)
- Conduct Final Security Review: At the end of the sprint, once you builded your software, you should look at all those tools output to spot security problems that may have occured, security regression and potential problems.
- Certify Archive and Release: Once you’re satifisfied and the software has been deployed sucessfully, do not destroy all the documentation but archive it. It may be enclosed in your project management tool, versionned using your favorite SCM or simply put in a folder. I STRONGLY advise you NOT to put the security documentation in your code base repository… well because it’s stupid and inefficient… it may contain ongoing security vulnerabilities and should be kept out of reach.
- (EXTRA) Security Unit testing: This one is not on the Agile SDL but does make sense to me. As much as possible you should try to integrate security scenarios into your unit testing. For example instead of testing regular input, test SQL injection or XSS attack or even fuzzing attack on your functions and assess that the output is as desired and not flawed. You may therefore use it for non-regression tests and can be used as security metrics all over your project.
- Create Quality Gates / Bugs bars: Define minimum acceptable levels of security and privacy quality. List the known vulnerabilities with priorities… Do be honest I didn’t get this one. For me either you iterate over requirements or add to Definition of Done but creating levels of security once in a while over development process doesn’t make sense… I recommend you do not care about this one.
- Perform dynamic analysis: Dynamic analysis is testing your software at runtime to look for vulnerabilities. Truth is it takes times and requires more complex tools and skills. That goes for fuzz testing too. Therefore I advise smaller companies to invest once in a while in pentesting consultants to assess this dynamic analysis and fuzz testing.
- Perform Fuzz Testing: As I said for dynamic analysis, doing pentesting assessment once in a while will do the trick.
- Conduct Attack Surface Review : Reviewing attack surface measurement upon code completion helps ensure that any design or implementation changes to an application or system have been taken into account, and that any new attack vectors created as a result of the changes have been reviewed and mitigated including threat models. I do not recommend to implement this one because most of the time the results of the tools or pentest would have given you enough information to detect new attack surface. It would be better to do a security review instead.
- (EXTRA)Security Level Assessment : Once again it’s not in Agile SDL but at this stage I use OWASP ASVS. Prior to each big release, I run a complete ASVS run to assess the security level of the application. You can find a cheatsheet to perform this assessment in my previous post.
- Establish Security Requirements: This one should be about general security risks that your application may encounter. You should determine if your application is a critical,medium or low in terms of security risk (high–banking system, medium–dealing with customer data application, low–displaying cat’s photos….) In this part you should also create a “security vulnerability” category in your issue tracking system.
- Perform Security and Privacy Risk Assessments: For me this part is redundant with the previous one. Once you established your requirements , iterate over it and adapt the security level. If you add sensitive data to your application, review your privacy and security requirements.
- Establish Design Requirements: This is purely about software design and choosing the most secure way to create software. Try to start from your requirement to see if current design matches. There is no silver bullet on this one, learn clean secure coding, prevent racing conditions and malicious component communication is really difficult.Just keep security in mind when you design new functionnalities.
- Perform Attack Surface Analysis/Reduction: Once your design is over, dedicate one brainstorming session to find the possible flaws or evil user stories to attack your software. Invite all developers and persons involved to have the widest view possible. Once done, integrate the feedback to review your design once more and bulletproof it for these scenarios.
- Create an Incident Response Plan: One day.. something will go wrong. Of course I don’t wish you but eventually you’ll need a backup plan in case you get hacked. Most of companies don’t own a incident response plan and it’s not always necessary but you can ask yourself simple question for security matters. For example, if your database with customer information are stolen and you discover it? What will you do ? Inform customer ? Regenerate all passwords? Will you suspend money transfers ? You can brainstorm with your team on those scenarios and document the results as a global policy that you would submit to your CEO or CTO. Such decisions have critical impact on business and that’s why they are important and should be discussed. However spending and documenting a full-feature document is useless most of the time for smaller companies.
That’s it ! If you have comments, if you agree or disagree, please let me know, I’d happily discuss with anyone having different opinions 🙂 I’m currently working on an alternative methodology to integrate security to agile processes, I’ll hope it will be useful for some of you!