diff --git a/.github/workflows/markdown-check-links.json b/.github/workflows/markdown-check-links.json deleted file mode 100644 index f636c2ac..00000000 --- a/.github/workflows/markdown-check-links.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "aliveStatusCodes": [200, 403] -} diff --git a/.github/workflows/markdown-check-links.yml b/.github/workflows/markdown-check-links.yml deleted file mode 100644 index 6ffcf3d8..00000000 --- a/.github/workflows/markdown-check-links.yml +++ /dev/null @@ -1,20 +0,0 @@ -name: Check Markdown Links - -on: - push: - branches: [main] - pull_request: - types: [opened, synchronize, reopened, ready_for_review] - branches: [main] - -jobs: - markdown-check-links: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - uses: gaurav-nelson/github-action-markdown-link-check@d53a906aa6b22b8979d33bc86170567e619495ec # 1.0.15 - with: - use-quiet-mode: "yes" - config-file: .github/workflows/markdown-check-links.json diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4291d795..6d933941 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,7 +4,7 @@ Thank you for your interest in improving the framework. In this guide you will get an overview of the contribution workflow from opening an issue, creating a PR, reviewing, and merging the PR. -The framework is concerned with engineering best practice at NHSD, however, operational matters (e.g. structure of teams, roles and responsibilities, etc.) are out of scope. Please consider this when adding new content, and if in doubt, any of the code owners will be more than happy to discuss. +The framework is concerned with engineering best practice, however, operational matters (e.g. structure of teams, roles and responsibilities, etc.) are out of scope. Please consider this when adding new content, and if in doubt, any of the code owners will be more than happy to discuss. If you're new to Github and/or Markdown, Github's own [contributor's guide](https://github.com/github/docs/blob/main/.github/CONTRIBUTING.md) provides good signposting on these topics. @@ -23,6 +23,7 @@ When committing your changes, please reference the issue they're resolving, so i * Preview your Markdown code to make sure the format is not broken. * Check grammar, spelling and punctuation, no one wants to look pedantic by requesting changes due to typos or inconsistent grammar/syntax, but it's only fair to keep this tidy. * The framework is open to the world. This has a few implications: - * Nothing in it should be confidential, private to NHSD or include any personal data. + * Nothing in it should be confidential, private to NHSE or include any personal data. * All links in framework pages should be public. + * Consider comments and Readme text: comments in open source repos such as this are not official communications and don't go through the normal approval process for public communication. They can however still be interpreted as official communication, even if not intended as such. For further details please see ["Consider comments and Readme text" in "Create knowledge"](principles.md#3-create-knowledge) * Use [inclusive language](inclusive-language.md): avoid terms which cause hurt and offence, including if they have historically been considered industry-standard terms. diff --git a/README.md b/README.md index 083f9643..72febd26 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # Software Engineering Quality Framework +> **Note:** Contributors must take all necessary precautions to ensure that sensitive data does not leak into Source Control Management Systems. You must ensure you have reviewed the [guidance on what to do should this happen](practices/guides/commit-purge.md). + ## How to get started See the [Quick Start Guide](quickstart.md) @@ -40,13 +42,8 @@ The framework is a companion to: The framework consists of: -* [Engineering principles](principles.md) +* [Engineering principles](principles.md) and [blueprints](blueprints.md) * [Engineering quality review tool](insights/review.md) -* [Communities of practice guidelines](communities/communities-of-practice.md) and active communities: - * [Product Development Test Automation Working Group](communities/pd-test-automation-working-group.md) - * [Product Development Engineering CoP](communities/pd-engineering-cop.md) - * [Product Development Cloud PaaS Working Group](communities/pd-cloud-working-group.md) - * [Secure Engineering CoP](communities/security-cop.md) * Guidance on how to practice [continuous improvement](continuous-improvement.md) ## Contributing diff --git a/blueprints.md b/blueprints.md new file mode 100644 index 00000000..2cdd6531 --- /dev/null +++ b/blueprints.md @@ -0,0 +1,14 @@ +# Engineering blueprints + +This is a list of blueprint solutions to common problems which are referenced within this [quality framework](README.md). + +Where possible this will be a set of fully working components / solutions you can use. Where that's not possible, it will be instructions. + +| Topic | Type of blueprint | Classification | Status | +| :-------------------------------------------------------------------------------------------------------------------------------- | :---------------- | :------------- | :-------- | +| [Creating GitHub repositories](https://github.com/nhs-england-tools/repository-template) | Full solution | Recommended | Published | +| [Automating versioning and releases for fast and secure flow](https://github.com/nhs-england-tools/versioning-reference-template) | Full solution | Recommended | In review | +| [Purging commits on GitHub](practices/guides/commit-purge.md) | Instructions | Mandatory | Published | +| [Signing commits on GitHub](practices/guides/commit-signing.md) | Instructions | Recommended | Published | +| [Automating performance-test decisions using APDEX](practices/performance-testing.md) | Instructions | Recommended | Published | +| [Scanning source code for secrets](tools/nhsd-git-secrets/README.md) | Full solution | Recommended | Published | diff --git a/communities/communities-of-practice.md b/communities/communities-of-practice.md deleted file mode 100644 index 9653ba58..00000000 --- a/communities/communities-of-practice.md +++ /dev/null @@ -1,164 +0,0 @@ -# Communities of practice - -This is part of a broader [quality framework](../README.md) - -A community of practice (CoP) is a group of people who "share a concern or a passion for something they do and learn how to do it better as they interact regularly" ([Introduction to communities of practice](https://wenger-trayner.com/introduction-to-communities-of-practice/)). This document gives guidance on forming and running a CoP. - -As Emily Webber, author of [Building Successful Communities of Practice](https://www.goodreads.com/book/show/29155800-building-successful-communities-of-practice), says: -> Connecting with other people, finding a sense of belonging and the need for support are natural human desires. Employees who don't feel supported at work don't stay around for long — or if they do, they quickly become unmotivated and unhappy. At a time when organisational structures are flattening and workforces are increasingly fluid, supporting and connecting people is more important than ever. This is where organisational communities of practice come in. - -# Community - -Members of a CoP have a shared enthusiasm and commitment to a particular topic. They engage in joint activities and discussions, help each other and share information. Through this they build relationships that enable them to learn from and gain support from each other. A CoP can become a supportive community which members reach out to for advice and which builds confidence. The community can become something members identify with and care deeply about, which creates a sense of belonging and purpose. Active participation in a CoP can also be a great way for individuals to raise their profile and build relationships with people in other areas of the organisation. - -# Learning - -A primary aim of most CoPs is learning, which typically happens in two ways: -1. Members share knowledge and experiences — good and bad. -1. Members stimulate each others' interest in learning from outside the organisation. - -Learning from outside the organisation includes discovering new tools and techniques from reading books or blogs, watching or participating in external events and so on. Having a community to share new ideas with encourages individuals to seek out new knowledge and to experiment with the new things they have learned. - -For individuals, this learning can become an important part of their personal and professional development. And the organisation benefits from having individuals who are more engaged and effective in their roles. - -# Scope - -CoPs are formed around a single subject which should be neither so narrow as to give little value to members (e.g. unit testing), nor so broad that it is difficult to find topics within the subject which are of broad interest to the group (e.g. all of software engineering). We discuss scope more in [Format](#format), as the scope can have an influence on the most appropriate format. As well as being of interest to members, to attract investment (see [Time](#time)) the subject should also be of relevance to the goals of the organisation, e.g. testing would be an appropriate subject for an organisation which delivers software, basket weaving would not, public speaking may be. - -# Goals - -It is important to set goals when forming a CoP to avoid it losing purpose and direction. Goals should be valuable to individuals and the wider organisation. - -Not all goals need to produce output in a material sense. Some focus more on harder to quantify results like individual learning. Having a balance of these is quite healthy and encouraged. If attendees start to see the CoP as just more actions to add to their to do list they could disengage. On the other hand, without clear goals a CoP can lose focus and can end up being little more than "a chat". When this happens, busy members can feel they don't get enough out of attendance to justify pulling themselves away from their busy schedules and again can disengage. It is a delicate balance to strike! - -Some common goals are listed below. We use the term _practitioner_ for someone whose primary job or role aligns strongly with the CoP’s scope, _apprentice_ for those who view the subject as an outside interest, _other_ for everyone else in the organisation and _public_ for anyone outside the organisation. - -* **Support** individual _practitioners_ with the specific challenges they are facing, usually with advanced or difficult topics. - -* **Support** individual _apprentices_ with the challenges they are facing, often with more basic techniques. - -* **Educate** by broadening and deepening _practitioners’_ and apprentices’ knowledge of the subject. - -* **Educate** by helping _practitioners_ and _apprentices_ learn about new and emerging tools and techniques. - -* **Educate** to help _others_ do their job better, typically for topics which are useful for all but not of sufficient interest to attend a CoP (e.g. writing user stories, load testing). - -* **Evaluate** projects, teams or individuals using specific knowledge of the subject, e.g. to develop and refine the way that _practitioners_ are assessed for recruitment and progression, or teams maturity is assessed to understand where outside support may be needed. - -* **Evangelise** to improve _public_ awareness of the organisation’s capabilities in this subject, usually to drive recruitment or customer acquisition by building credibility. - -It may be beneficial to focus on one or two goals initially with the intention to take on other goals over time. A common challenge which CoPs encounter is maintaining focus to deliver useful outcomes with the limited time available (see [Time](#time)). The best way to avoid this is to set achievable goals rather than being overambitious. - -# Format - -CoPs are fuelled by interactions between individuals. Broadly, interactions are either "live" (either in person or remotely by video call) or asynchronous (recorded or written). Several interaction formats are common, and CoPs often use more than one of these: - -* Some form of regular meeting (default monthly) is usually the cornerstone of a CoP. These can take different formats at different times, e.g. - * Live "round table" or [lean coffee](https://leancoffee.org/) format discussion. This works well for small groups (5 to 10 people) to provide structure without imposing an agenda. - * Meeting with a specific topic or purpose and potentially an agenda. This is good when the CoP has identified an area they want to focus on or an outcome they want to achieve, but highly structured sessions like this are best used sparingly. - -* Internal or public "clinic". - -* Chat channel, e.g. in MS Teams or Slack. - * This does not constitute the community nor replace regular meetings, but it encourages members to keep interacting with each other - -* Live or recorded training sessions. - -* "Dojos" - * A hands-on practical session to try out specific new or unfamiliar techniques or tools in a safe environment. - * Works well for small groups (5 to 10 people) to learn how to use a specific tool or technique in a practical and quick way. - -* Social events. Getting members together in a more social setting can help people to get to know each other and create a sense of belonging and community. - -* Live or recorded show and tell or presentation — for internal or public audience. - -* Written show and tell or presentation (blog post style — internal or public). - -* Shared written examples (e.g. of definitions of done or test strategies). Usually in a raw state and for internal use only. - -* Written training material for internal or public use. - -* Written policies or standards — for internal use, but could still be public. - -Different formats are better suited to different goals, depending on: - -* How repeatable the communication needs to be, e.g. providing bespoke support vs a set of interview questions. - -* The interest level of the audience, from practitioner to public. - -* The amount of information to be communicated, e.g. a lessons learned presentation vs a policy or standard. - -Think carefully about which format or combination of formats will best suit the particular goal in mind. - -# Leadership - -Well-functioning CoPs do not come about by accident, and good leadership is essential to establish clear goals and the operating model. The aim is not to direct or control the CoP, but to provide the support and focus needed to create common purpose and expectations in the group and to evolve the goals and format over time. - -Typically, CoPs which work well have a designated coordinator plus a core group. The coordinator can be a rotating post, e.g. with a different person being appointed every three months. The core group is simply the members who most actively and regularly participate and contribute. - -# Time - -CoPs provide valuable learning opportunities, which benefit individuals and the organisation. As such, it is often appropriate to run CoPs over lunch time or after work to avoid impacting the working day. Many CoPs will provide refreshments of some sort to recognise and encourage the personal commitment from participants — anything from biscuits to sandwiches or pizza are common. - -However, experience has shown that CoPs are most effective when at least some "work" time is allocated also. This is especially necessary where preparation is necessary (e.g. for presentations), for those coordinating CoPs, or where the goal is more clearly to deliver benefit for the organisation than the individual (e.g. creating interview questions, assessment frameworks or policies and standards). And in areas where the organisation wishes to place particular focus (e.g. security) it may be appropriate to hold regular meet ups in work hours. - -Time can be allocated using a combination of a per person budget and a per CoP budget: - -* The **per person budget** provides rough guidance on an appropriate amount of "work" time for each individual to spend on CoP activity across all CoPs they participate in. It should be personalised and agreed with the individual’s manager or equivalent. The appropriate amount of time will naturally vary between individuals and will balance benefit to the individual and the organisation from participating in CoPs with the demands of the individual’s primary role. This is the main mechanism to regulate the cost of recurring sessions such as monthly meet ups. - -* The **per CoP budget** is a monthly time allocation for the CoP to be distributed by the coordinator to members of the group and is on top of the per person budget. The amount of time should be appropriate to deliver the agreed goals but would typically be a few hours to a few days per month. This allocation could vary over time and would be higher for periods where there is particular focus on delivering goals which will take more time commitment. - -# Sponsor - -Any allocation of "work" time for CoP activity or cost associated with providing refreshments or materials (e.g. software licences) for the CoP will need to be approved by someone. This person is called the _sponsor_ and should be invested in the goals of the CoP and authorised to allocate these resources. - -# Starting a new CoP - -The process for starting a new CoP is: - -1. Briefly outline the subject to be covered (see [Scope](#scope)), a set of [goals](#goals) and a proposed [time](#time) allocation and agree these with the [sponsor](#sponsor). - -2. Appoint a [coordinator](#leadership) and agree whether this will be a rotating post. The coordinator should be someone with the time and energy to actively drive the community, and the ability to excite and enthuse members. They need to be able to provide just enough direction and structure while ensuring members individual points of view are clearly heard and respected. - -3. The coordinator will facilitate a kick off along with interested others where the format will be agreed. The goal here is to get people excited and engaged and to align members so they can act as an effective group. - -4. As the community forms they should refine the scope, goals and format along with the coordinator and sponsor to ensure they feel ownership of the community and to gain the benefit of their experience and knowledge. These should be recorded somewhere easily accessible to the whole organisation, and easily editable by CoP members, e.g. Confluence. - -# Worked example -We include a worked example to illustrate how this thinking can be applied in a specific case. This is an example of a new CoP which is starting with relatively modest goals and time budget. - -> #### Subject -> AWS -> -> #### Sponsor -> Stacy Leblanc -> -> #### Goals -> These will be the initial goals to be reviewed after three months on _DATE_. -> -> * Support individual practitioners and apprentices with specific challenges. -> -> * Educate by broadening and deepening practitioners’ and apprentices’ knowledge of the subject. -> -> * Educate by helping practitioners and apprentices learn about new and emerging tools and techniques. -> -> #### Scope -> This group will focus on AWS native services. AWS marketplace services are out of scope for this group. -> -> #### Coordinator -> This will be a rotating post. Seren Howarth will be coordinator for the first three months. -> -> #### Format and time -> The group will meet monthly for one hour over lunch, where food will be provided (up to £6 per head). The coordinator will help the group determine the format for the next session each time. A time budget of two hours per month (on average) will be used to prepare for sessions when required. - -# Tips for success - -* Make the CoP visible. Help potential members easily understand the purpose of the CoP and what it is doing, as well as how it relates to other CoPs or working groups. Most importantly, focus on what they can get out of it! Regular communications using a variety of channels (e.g. MS Teams, email, mentions at all hands calls) can help build awareness and interest. Public endorsement from the sponsor will increase credibility and authority. -* Use the tools you know. Many of the tools we use for managing our day to day work can help CoPs, including group messaging systems, places to store knowledge (e.g. Wiki) and hold a lightweight backlog of things the CoP wants to work on (e.g. Jira, MS Teams Planner). -* Individuals must be willing and keen to participate. Participation should be voluntary. In this way, Communities of Practice live on because they create value for members, not because of an edict or a box to check. Individuals can be encouraged to join, but it must be their free choice to avoid the CoP becoming just another meeting to go to. -* Be realistic about what can be achieved with the time available. Members will likely be willing to spend an occasional lunch time for CoP meetings if they get value from them, but where tasks are picked up to be done between meetings, more will be achievable if there is some allocated time, rather than it being down to "best endeavours". -* Keep it interesting. Try different formats and play with different ideas. Find out what other CoPs are doing and try things which seem interesting. -* Focus on people. CoPs are there to serve people, not to act as another project. Be mindful of their needs and devote time to serving them, which may include acting as a support network. -* But also, focus on outcomes. Get the balance right between serving individuals and their immediate needs and producing wider and longer term benefits for the organisation — particularly if work time is being allocated to CoP activities. -* Don't try to solve everything. While the CoP can be a great focal point, ideally it will put in place mechanisms for- and champion peer to peer knowledge sharing and support throughout the organisation, rather than expecting everything to flow through the CoP. -* Individuals should continue to get value from their participation. Meetings do not need to be formal or highly structured, but participants should feel that it is productive use of their time. Hold retrospectives periodically to ensure the community is still delivering value and to refresh the goals and ways of working. diff --git a/communities/pd-cloud-working-group.md b/communities/pd-cloud-working-group.md deleted file mode 100644 index 5e9367b9..00000000 --- a/communities/pd-cloud-working-group.md +++ /dev/null @@ -1,50 +0,0 @@ -# NHS Digital Cloud PaaS Working Group (CWG) - -## Subject -All aspects of cloud based activities within NHS Digital Cloud PaaS arena. - -## Sponsor -Andy Blundell ([andyblundell](https://github.com/andyblundell)) - -## Goals -* Provide a class leading cloud platform -* Technical Steer for Cloud - -## Purpose and format -* Discuss upcoming changes within NHSD Cloud PaaS Arena and service team responsibilities -* Highlight best practice -* Outline implementation plans and timescales -* Discuss priorities of NHSD Cloud PaaS platform work and backlog -* Review and input into NHSD Cloud PaaS platform Roadmap -* Resolve bottlenecks -* Review processes, new technologies and tools -* Training, mentoring and guidance (Show ‘n Tells, Demos, DevOps Sessions, Cloud Provider Sessions) -* CWG HackDays - -## Responsibilities -* Enhancement recommendations for NHSD Cloud PaaS Platform -* Accountable to NHSD Cloud Policies and NHSD Cloud PaaS Principles -* Input into the NHSD Cloud PaaS backlog and its Roadmap - -## Coordinator -Chaired by member of NHSD Cloud PaaS Team (Neal Lilly/Jonathan Hadley) - -## Members -All Service Teams onboarding/running in NHSD Cloud PaaS, recommendation of at least one representative from each service -Other Teams interacting with NHSD Cloud PaaS (e.g. NHSD Splunk Team) - -## Agenda -New Items can be raised in the agenda by Technical Leads - -## Frequency -Weekly - -## Actions -Recorded in Agenda, Actions raised as Jira Tickets - -## Review -* Cover previous open agenda items/Jira Tickets -* Goals to be reviewed Annually December 2020 - -## Related -Service Team Stand Up. NHSD Cloud PaaS Refinement. diff --git a/communities/pd-engineering-cop.md b/communities/pd-engineering-cop.md deleted file mode 100644 index f367d69d..00000000 --- a/communities/pd-engineering-cop.md +++ /dev/null @@ -1,51 +0,0 @@ -# Engineering CoP (Community of Practice) - -This is part of a broader [quality framework](../README.md) and is one of a set of [communities of practice](communities-of-practice.md). - -There is some overlap with the [Test Automation Working Group](pd-test-automation-working-group.md). - -## Subject - -All aspects of software engineering in NHS Digital Product Development. - -## Sponsor - -Andy Blundell ([andyblundell](https://github.com/andyblundell)) - -## Goals - -* Identify **good practices and patterns** (as well as those to avoid), and make these easily accessible for all engineers. - * This includes maintaining the [principles](../principles.md), patterns and practices in this repository. - * This may also include arranging talks or show and tell sessions provided either internally or by guest speakers. -* Provide clear guidance for engineers on which **tools and technologies** are (and are not) recommended, including: - * Playing an active role in maintaining the internal technology radar. - * Being aware of and assessing new and emerging technologies. -* Help individuals and teams **find solutions** to the specific engineering challenges they face. -* Maintain references to **learning resources** relevant to each part of the [Software Engineering Quality Framework](../README.md) - -Goals to be reviewed December 2020 - -## Scope - -Initially this CoP will cast a very wide net and cover front-end, back-end and platform/infrastructure engineering and all technical aspects of delivery and operation of reliable services. This will include topics like pair programming, effective code reviews, team quality processes, architecting and engineering for functional and non-functional requirements and validating they are met, e.g. performance and capacity, security, accessibility. Some aspects of this will be covered in more detail by the [Test Automation Working Group](pd-test-automation-working-group.md), while others are expected to spin out into their own CoPs over time in response to demand to provide more focus around specific sub-topics (e.g. mobile engineering, operational excellence, front-end development). - -## Coordinator - -* This will be a rotating post on a 3-month basis, requiring a commitment of one day per week. -* Ivor Caldwell ([ivorc](https://github.com/ivorc)) will be coordinator initially. -* TBC will be deputy coordinator (to cover sessions when the coordinator is away, etc) who will typically take over as coordinator for the following 3 months. - -## Members - -* The CoP will have a core group who work together to deliver the CoP goals for the wider engineering community. -* Core members will be ambassadors for the CoP, the [Software Engineering Quality Framework](../README.md) and the Tech Radar. -* Members will be expected to attend and contribute at the regular core group meetings. -* Occasional time commitment outside these meetings will also be necessary, e.g. to prepare a presentation or hack event or to help others to do so. - -## Format - -* The group will have an open backlog (location TBC). The Sponsor and Coordinator will manage the backlog. -* The group will meet every month to discuss issues, demonstrate solutions, and agree who will work on new tickets. -* To keep these meetings manageable and useful, the aim is to have one representative from each area (this could be a single person, or a rotating role). The aim is for between 10 to 20 people to attend each session. -* The group will also organise and facilitate other meetings, such as training workshops, hack-days, etc, which will be open to a much wider audience. These are expected to cover a representative spread across engineering disciplines, and as such each will likely appeal to a different subset of the wider community. -* The group will maintain an open discussion channel (location TBC) where the wider community can seek and share advice and guidance, and a restricted channel for the core group to collaborate together. There may be additional channels for sub-topics (e.g. mobile, front-end dev, aws). diff --git a/communities/pd-test-automation-working-group.md b/communities/pd-test-automation-working-group.md deleted file mode 100644 index 227daf9e..00000000 --- a/communities/pd-test-automation-working-group.md +++ /dev/null @@ -1,57 +0,0 @@ -# Test Automation Working Group - -This is part of a broader [quality framework](../README.md) and is one of a set of [communities of practice](communities-of-practice.md) - -## Subject - -Covers Test Automation and Quality Checking tools and approaches within NHS Digital Product Development. - -## Sponsor - -Andy Blundell [andyblundell](https://github.com/andyblundell) - -## Goals - -The goal is to provide solutions and guidance on Automated Testing and Quality, including: - -* Share knowledge -* Provide advice and guidance -* Deliver solutions to relevant issues -* Curate relevant Training Pathways within the [Software Engineering Quality Framework](../README.md) -* Provide supplementary training in relevant areas (e.g. guest speakers, workshops) -* Curate relevant principals, practices and patterns within the [Software Engineering Quality Framework](../README.md) - -Goals to be reviewed December 2020 - -## Scope - -Areas of interest are specifically: - -* Automated Functional Testing -* Automated Non-Functional Testing, e.g. load/capacity/performance/soak testing, security testing, etc -* Automated Code Hygiene. e.g. static code analysis, dead code detection, code profiling, technical debt, etc - -## Coordinator - -* This will be a rotating post on a 3-month basis, requiring a commitment of one day per week -* The coordinator will run a blog to help publicise the activity of the group -* There will be a deputy coordinator (to cover sessions when the coordinator is away, etc). They will also typically take over as coordinator for the following 3 months. This gives the group continuity. -* David Lavender [dlavender4](https://github.com/dlavender4) will be coordinator initially -* A Deputy will be found during the first sessions - -## Members - -* Formed from within PD -* Core members should be kept to under 15, to promote interactive sessions -* Should include a mix of backgrounds including Dev, DevOps, Test and Assurance. -* Should include people from a representative spread of teams -* Can join as representatives from a specific team, or as "interested parties" - -## Format - -* The group's official home is the "Test Automation CoP" Teams channel -* This includes a Backlog and Wiki -* Any member can contribute to the backlog. This could be show-and-tells of tools or approaches; queries about how to do something; curation of a principle; talking about an organisation-wide policy; etc. -* The group meet regularly to refine the backlog, and talk through items -* The group will also organise and facilitate other meetings, such as training workshops, hack-days, etc, which will be open to a much wider audience. -* The group will maintain a discussion channel ("Test Automation") open to all of PD: for advice and guidance, and wider knowledge sharing diff --git a/communities/security-cop.md b/communities/security-cop.md deleted file mode 100644 index fe9e64bf..00000000 --- a/communities/security-cop.md +++ /dev/null @@ -1,66 +0,0 @@ -# Secure Engineering CoP (Community of Practice) - -This is part of a broader [quality framework](../README.md) and is one of a set of [communities of practice](communities-of-practice.md) - -## Subject - -Secure Engineering practices, tools & approaches within NHS Digital. - -## Sponsor - -(TBC who) - -## Goals - -For secure development and operation of systems: - -* **Share** knowledge: - * Provide advice and guidance as requested - * Facilitate good practice discussions, and curate relevant principles, practices and patterns within the [Software Engineering Quality Framework](../README.md) - * Curate relevant Training Pathways within the [Software Engineering Quality Framework](../README.md) - * Curate relevant section(s) within the [Software Engineering Review Tool](../insights/review.md) - * Provide supplementary learning in relevant areas (e.g. workshops) - * Discuss, disseminate and feedback on the output of the Cyber Design Authority -* **Build** knowledge: - * Organise learning events (e.g. guest speakers) - * Organise events to practice security (e.g. security jams) - * Evaluate candidate security tools - * Form mini-groups to respond to "how do I..." questions from teams - generating examples in the [Software Engineering Quality Framework](../README.md) - -Goals to be reviewed after 3 months - -## Scope - -Areas of interest are specifically: - -* Secure development & operations practices -* Security good practice -* Automated security-testing tools, for example tools to scan for secrets or other sensitive data - -## Coordinator - -* Initially: - * There will be a faciliator group, comprising of a representative from each of the member NHS Digital directorates (DSC, Product Development, Platforms, Security Architecture) - * The group will self-organise who faciliates individual sessions & activities -* Once the community is established: - * This will be a rotating post on a 3-month basis, requiring a commitment of one day per week - * The coordinator will run a blog to help publicise the activity of the group - * There will be a deputy coordinator (to cover sessions when the coordinator is away, etc). They will also typically take over as coordinator for the following 3 months. This gives the group continuity. - -## Members - -* This group will span NHS Digital and include representatives from DSC, SSS, and delivery teams -* If possible, this group will also include external (to NHS Digital) subject-matter experts -* Core members should be kept to under 15, to promote interactive sessions -* Should include a mix of backgrounds including members of the Data Security Centre and team members with no Security qualifications -* Should include people from a representative spread of teams / directorates -* Can join as representatives from a specific team, or as "interested parties" - -## Format - -* The group's official home is (TBC - where) -* This includes a Backlog -* Any member can contribute to the backlog. This could be show-and-tells of tools or approaches; queries about how to do something; curation of a principle; talking about an organisation-wide policy; etc. -* The group meet regularly (TBC - how frequently) to refine the backlog, and talk through items -* The group will also organise and facilitate other meetings, such as training workshops, hack-days, etc, which will be open to a much wider audience. -* The group will maintain a discussion channel (TBC - where) open to all of NHS Digital: for advice and guidance, and wider knowledge sharing diff --git a/images/commit-history-github.png b/images/commit-history-github.png new file mode 100644 index 00000000..80a5574f Binary files /dev/null and b/images/commit-history-github.png differ diff --git a/images/cycle-through-flagged-commits.png b/images/cycle-through-flagged-commits.png new file mode 100644 index 00000000..85025296 Binary files /dev/null and b/images/cycle-through-flagged-commits.png differ diff --git a/images/interactive-rebase-after-selecting-commits.png b/images/interactive-rebase-after-selecting-commits.png new file mode 100644 index 00000000..dfada4e9 Binary files /dev/null and b/images/interactive-rebase-after-selecting-commits.png differ diff --git a/images/interactive-rebase-before-selecting-commits.png b/images/interactive-rebase-before-selecting-commits.png new file mode 100644 index 00000000..c32879f6 Binary files /dev/null and b/images/interactive-rebase-before-selecting-commits.png differ diff --git a/images/merging-is-blocked-signatures.png b/images/merging-is-blocked-signatures.png new file mode 100644 index 00000000..c3dd9bd5 Binary files /dev/null and b/images/merging-is-blocked-signatures.png differ diff --git a/images/updated-commit-history-github.png b/images/updated-commit-history-github.png new file mode 100644 index 00000000..f9ecd05d Binary files /dev/null and b/images/updated-commit-history-github.png differ diff --git a/inclusive-language.md b/inclusive-language.md index d0856ca6..42b14c12 100644 --- a/inclusive-language.md +++ b/inclusive-language.md @@ -10,7 +10,7 @@ ## Context - This is part of a broader [quality framework](README.md) -- This guidance has been co-authored with the NHS Digital [EMBRACE network](https://digital.nhs.uk/about-nhs-digital/corporate-information-and-documents/staff-networks#ethnic-minorities-broadening-racial-awareness-and-cultural-exchange-embrace-) and the [Lesbian, Gay, Bisexual, Transgender, Queer and Allies (LGBTQ+) network](https://digital.nhs.uk/about-nhs-digital/corporate-information-and-documents/staff-networks#lesbian-gay-bisexual-transgender-queer-and-allies-lgbtq-) +- This guidance has been co-authored with the NHS Digital EMBRACE network and the [Lesbian, Gay, Bisexual, Transgender, Queer and Allies (LGBTQ+) network](https://www.england.nhs.uk/about/working-for/staff-networks/) ## Background diff --git a/insights/review.md b/insights/review.md index 8afda90d..9ac3f938 100644 --- a/insights/review.md +++ b/insights/review.md @@ -167,7 +167,11 @@ You may wish to score each individual component or system separately for these a #### 9. Testing - We have great test coverage. -- Testing is everyone's responsibility. +- Testing is everyone's responsibility and test is a first-class concern. +- A failing test suite in CI gets immediate attention. +- We support all team members to practice good testing, including by holding no-blame sessions to discuss any automation tests we should have added, and what we can learn from having missed them initially. +- We build code for testability. +- Tests (including both test code and test coverage & whether there are gaps) are part of our standard peer-review process. - Repetitive tests are automated. - Testing is considered before each work item is started and throughout its delivery. - We use the right mix of testing techniques including automated checks and exploratory testing. diff --git a/patterns/deployment.md b/patterns/deployment.md index 53e55af3..92ffe55f 100644 --- a/patterns/deployment.md +++ b/patterns/deployment.md @@ -16,7 +16,7 @@ When developing services and applications development teams must consider their This is a fundamental step in the design of the solution, development teams must consider the technical impact of different strategies alongside the impact to service during deployments. This starts at the point of building the CI pipelines and permeates through to final live deployment pipelines. Different approaches may be applicable depending on the chosen approach, for example serverless deployments as opposed to VM based deployments. -Deployments must be repeatable and [idempotent](../practices/continuous-integration.md#deploy-what-you-tested), so that deploying the same version twice will result in the dame deployed environment. +Deployments must be repeatable and [idempotent](../practices/continuous-integration.md#deploy-what-you-tested) so that deploying the same version twice will result in the same deployed environment. ## CI/CD pipeline-based deployment @@ -120,9 +120,9 @@ The following table includes steps that development teams should consider when p ### Failure mode -If initial smoke tests fail OR monitoring identifies increased failures / other indicator OR Further smoke tests fail: +If initial smoke tests fail OR monitoring identifies increased failures / other indicators OR Further smoke tests fail: -1. Traffic is migrated back to previously healthy leg. +1. Traffic is migrated back to the previously healthy leg. 1. Release is marked as failed. ### Game days and chaos testing diff --git a/patterns/everything-as-code.md b/patterns/everything-as-code.md index b8a4ce21..664c1467 100644 --- a/patterns/everything-as-code.md +++ b/patterns/everything-as-code.md @@ -46,7 +46,7 @@ A code review involves another member of the team looking through a proposed cod Many teams consider code which has been written [as a pair](https://martinfowler.com/articles/on-pair-programming.html) to already have been reviewed, and do not require a separate review. -Robert Fink provides an excellent description of the [motivation and practice of code reviews](https://medium.com/palantir/code-review-best-practices-19e02780015f). Some key points from this and other sources ([Google](https://google.github.io/eng-practices/review/reviewer/), [SmartBear](https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/), [Atlassian](https://www.atlassian.com/agile/software-development/code-reviews)) are: +Robert Fink provides an excellent description of the [motivation and practice of code reviews](https://blog.palantir.com/code-review-best-practices-19e02780015f). Some key points from this and other sources ([Google](https://github.com/google/eng-practices/blob/master/review/reviewer/index.md), [SmartBear](https://smartbear.com/learn/code-review/best-practices-for-peer-code-review/), [Atlassian](https://www.atlassian.com/agile/software-development/code-reviews)) are: #### Egalitarian @@ -70,6 +70,9 @@ While effective testing is the best way to detect bugs or non-functional problem - Is the code clear and simple? - Is the code layout and structure consistent with agreed style and other code? (please see [enforce code formatting](enforce-code-formatting.md)) - Would it easily allow future modification to meet slightly different needs, e.g. ten times the required data size or throughput? +- Is it [built for testability](../practices/structured-code.md)? +- Are the automated tests positioned appropriately in the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html), triggered appropriately in CI builds, and do they block the build when they fail? +- Are there any missing [automated tests](../practices/testing.md), e.g. edge-cases that have not yet been considered? - Have the non-functional requirements been considered (performance, scalability, robustness, etc)? - Are common security issues guarded against (e.g. [OWASP Top 10](https://owasp.org/www-project-top-ten/))? Including: - Is any new input data being treated as potentially hostile? diff --git a/patterns/little-and-often.md b/patterns/little-and-often.md index 9e2616a1..204a028f 100644 --- a/patterns/little-and-often.md +++ b/patterns/little-and-often.md @@ -48,7 +48,7 @@ This pattern is in essence very straightforward; the power comes in applying it - **Delivering software.** The trivial and obvious example of this pattern is that it is better to deliver software in small increments than to do it in one [big bang](https://hackernoon.com/why-your-big-bang-multi-year-project-will-fail-988e45c830af). - **Planning.** Start by doing just enough planning to forecast the size and type of team(s) you need to get the job done roughly when you want it to be done by. Incrementally refine that plan through (typically) fortnightly backlog/roadmap refinement sessions. -- **User-centred design.** User research and design activities ([SERVICE-USER](https://www.gov.uk/service-manual/user-research), [SERVICE-DESIGN](https://www.gov.uk/service-manual/design)) occur in all phases of an agile delivery: [discovery](https://www.gov.uk/service-manual/agile-delivery/how-the-discovery-phase-works), [alpha](https://www.gov.uk/service-manual/agile-delivery/how-the-alpha-phase-works), [beta](https://www.gov.uk/service-manual/agile-delivery/how-the-beta-phase-works) and [live](https://www.gov.uk/service-manual/agile-delivery/how-the-live-phase-works) ([SERVICE-PHASES](https://www.gov.uk/service-manual/agile-delivery)). Delivery in all phases is done using [build-measure-learn](http://theleanstartup.com/principles#:~:text=A%20core%20component%20of%20Lean,feedback%20loop.) loops, with the whole multi-disciplinary team working closely together in all three activities. This approach means that rather than having a big up front design, the design is iteratively refined throughout all phases ([SERVICE-AGILE](https://www.gov.uk/service-manual/agile-delivery/agile-government-services-introduction#the-differences-between-traditional-and-agile-methods)). +- **User-centred design.** User research and design activities ([SERVICE-USER](https://www.gov.uk/service-manual/user-research), [SERVICE-DESIGN](https://www.gov.uk/service-manual/design)) occur in all phases of an agile delivery: [discovery](https://www.gov.uk/service-manual/agile-delivery/how-the-discovery-phase-works), [alpha](https://www.gov.uk/service-manual/agile-delivery/how-the-alpha-phase-works), [beta](https://www.gov.uk/service-manual/agile-delivery/how-the-beta-phase-works) and [live](https://www.gov.uk/service-manual/agile-delivery/how-the-live-phase-works) ([SERVICE-PHASES](https://www.gov.uk/service-manual/agile-delivery)). Delivery in all phases is done using [build-measure-learn](https://theleanstartup.com/principles#:~:text=A%20core%20component%20of%20Lean,feedback%20loop) loops, with the whole multi-disciplinary team working closely together in all three activities. This approach means that rather than having a big up front design, the design is iteratively refined throughout all phases ([SERVICE-AGILE](https://www.gov.uk/service-manual/agile-delivery/agile-government-services-introduction#the-differences-between-traditional-and-agile-methods)). - **Technical design and architecture.** While some up front thinking is generally beneficial to help a delivery team set off in the right direction, the output design is best viewed as first draft which will be refined during delivery as more is discovered about technical and product constraints and opportunities. See [Evolutionary Architectures](https://evolutionaryarchitecture.com/precis.html). - **Team processes.** Great team processes come about by starting with something simple and practising [continuous improvement](https://kanbanize.com/lean-management/improvement/what-is-continuous-improvement) to find ways of working, definitions of done and so on which are well suited to the particular team and environment. diff --git a/patterns/outsource-bottom-up.md b/patterns/outsource-bottom-up.md index 4b93ddb8..573151d3 100644 --- a/patterns/outsource-bottom-up.md +++ b/patterns/outsource-bottom-up.md @@ -3,9 +3,12 @@ - [Outsource bottom up](#outsource-bottom-up) - [Context](#context) - [The pattern](#the-pattern) - - [Benefits](#benefits) + - [Benefits](#benefits) + - [Tradeoffs](#tradeoffs) - [Details](#details) - - [Cloud native vs cloud agnostic](#cloud-native-vs-cloud-agnostic) + - [Containers vs cloud functions](#containers-vs-cloud-functions) + - [Data persistence](#data-persistence) + - [Certificates](#certificates) - [Caveats](#caveats) ## Context @@ -19,31 +22,59 @@ Use managed services where available and appropriate. The aim is to reduce operational burden by handing responsibility to the cloud provider. They have made a business from doing this better than most organisations can. -## Benefits +### Benefits -- Reduced effort in building the "infrastructure" makes more time to build business valuable functionality. +- Reduced effort in building the "infrastructure" makes more time to build valuable business functionality. - Reduced operational maintenance overhead. - Improved reliability. - Improved security. +### Tradeoffs + +- Vendor lock-in +- Reduced technical choice +- New platforms to learn +- Less familiar development process + +Our bet is that the benefits outweigh the tradeoffs. On a spectrum between self-managed and entirely vendor-managed, solutions with more vendor management will tend to lock you in more, and make any future platform change harder. In return, we hope for a smaller solution overall, and getting new platform features faster. Similarly the reduced technical choice in language runtimes, data infrastructure, or observability is compensated for by the support levels that the vendor-supplied options enjoy. + +Sometimes we have a technical or legislative need to be closer to the self-managed end of the spectrum, and sometimes the ways of working necessary to use cloud products do not align with other engineering requirements, but our first instinct should be to offload responsibility for as much as possible to the vendor. + +Some cloud-native choices inevitably imply specialising our solution for that one technology, which could create a degree of vendor lock-in. For instance, for a particular use case AWS Lambda may allow you to deliver and operate far more cheaply than Kubernetes, but would require work to re-engineer if moving to a different cloud provider. + +You need to understand and be able to justify a choice that creates vendor lock-in. If the benefits of vendor lock-in outweigh the pain of being locked in, then being locked in is a good thing. Similarly, when looking at the other side of the coin bear in mind that *any* migration between providers, whether from vendor-specific technology or generic, will be expensive because - at the very least - of data locality, access management, and uptime considerations. + +The choice is between investing time early to build in mobility and deferring that effort (quite possibly forever) to focus on things which deliver more value to the organisation, but following the principle that nothing untested can be assumed to work, engineering principles dictate that you should have a plan to continue to assure that mobility as the project evolves. If mobility is a required feature, that feature must be tested. + ## Details -- Prefer software as a service (SaaS, e.g. Splunk, Jira), then serverless platform as a service (PaaS, e.g. Amazon DynamoDB, AWS Lambda), then infrastructure as a service (IaaS, e.g. cloud VMs). -- For compute, prefer functions as a service (e.g. AWS Lambda), then serverless containers (e.g. GKE, AWS Fargate), then VM based deployments (e.g. AWS EKS, AWS ECS on EC2). - - TO DO: close discussion about whether containers are still preferable in some use-cases -- For data persistence prefer (where there are no other differentiating factors) pay per request options (e.g. Amazon DynamoDB, S3) to pay per time choices (e.g. Amazon Aurora or RDS). -- In general, prefer solutions which do not involve managing VMs if possible, and ideally where there is no explicit configuration of a network (e.g. subnets, internet gateways, NAT gateways) — compare AWS Lambda which needs no network with AWS Fargate which does. -- Where possible, outsource the management (including renewal) of certificates (e.g. via [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/)) - - Where that isn't possible, still prefer outsourcing the management of alerting for certificate expiry (see [observability](../practices/observability.md)) +Always prefer software as a service (SaaS, e.g. Splunk, Jira), then serverless platform as a service (PaaS, e.g. Amazon DynamoDB, AWS Lambda, Texas, AWS Fargate), then infrastructure as a service (IaaS, e.g. cloud VMs). Cloud VMs are the least flexible option available to us, so should only be used if no other option can be chosen. + +Similarly you should prefer solutions which do not require you to manage a network configuration. + +### Containers vs cloud functions + +If you have a choice as to whether to use functions as a service or a container platform for a part of your implementation, you should consider: + +- Are you migrating an existing service, or implementing something new? It may be easier to design a new service on cloud functions and cloud services than on a container platform. Migrating an existing service may be easier if first moved to containers, then having appropriate parts extracted to cloud functions at a later date. +- Is there a software framework particularly well suited to your problem that better fits containers, or to cloud functions? For instance, you may prefer the Django admin interface to having to build your own: this would be better suited to a container platform than on AWS Lambda. +- Implementations based on cloud functions can be harder to test, because you might not be able to instantiate service dependencies locally. Is that important for your use case? Do you have a good developer testing story otherwise? +- Is your implementation primarily plumbing between other services already in the cloud, with little logic of its own? Is it primarily asynchronous? A "yes" to either of these would point in the direction of cloud functions. +- Do you have specific language or runtime version requirements? While these can usually be accommodated on cloud function platforms, there is additional work if your needs do not align with the platform-supplied runtimes. You may find that this erases any advantage to cloud functions over containers. + +### Data persistence + +For data persistence, where there are no other differentiating factors, prefer pay per request options (e.g. Amazon DynamoDB, S3) over pay per time choices (e.g. Amazon Aurora or RDS). + +Bear in mind that for higher-throughput applications where the usage would never scale to zero, or where cold start delays would be problematic, the difference between an autoscaling pay-per-time choice and serverless pay-per-request model may tilt in the other direction. The correct choice will depend on the differentiating factors of your specific application. + +A more fundamental principle is that you should never send a document store to do a relational job. Don't pick a NoSQL implementation when you need an RDBMS just because of the payment model: any savings you might expect from the payment model will result in a more expensive, slower, and more painful development experience. -## Cloud native vs cloud agnostic +### Certificates -A closely related, but subtly distinct consideration is whether to prefer cloud native (serverless) technologies or cloud agnostic / generic ones. Cloud native choices inevitably imply specialising our solution for that one technology, which could create a degree of vendor lock in. +Where possible, outsource the management (including renewal) of certificates (e.g. via [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/)) -- Understand and be able to justify vendor lock in. - - If the benefits of vendor lock in outweigh the pain of being locked in, then being locked in is a good thing. - - e.g. for a particular use case AWS Lambda may allow you to deliver and operate far more cheaply than Kubernetes, but would require work to re-engineer if moving to a different cloud provider. - - The choice is between investing time early to build in mobility and deferring that effort (quite possibly forever) to focus on things which deliver more value to the organisation. +Where that isn't possible, still prefer outsourcing the management of alerting for certificate expiry (see [observability](../practices/observability.md)) ## Caveats @@ -51,3 +82,4 @@ A closely related, but subtly distinct consideration is whether to prefer cloud - Performance tuning can be needed to achieve reliable “flat” response times both during very quiet and very busy periods. - Special efforts may be needed to ensure services are "kept warm" at quiet times. - At busy times, throttling or buffering throughput may be needed to avoid overwhelming any related systems which do not scale as elastically. +- Do not assume that services calling yours will throttle or buffer. Build in load shedding and rate limiting to protect your own service where needed. While we aim to be good neighbours of our upstream APIs, failing gracefully in response to excess traffic is your responsibility. diff --git a/practices/actions-best-practices.md b/practices/actions-best-practices.md new file mode 100644 index 00000000..350203e4 --- /dev/null +++ b/practices/actions-best-practices.md @@ -0,0 +1,238 @@ +# GitHub Actions Security Best Practices + +## Introduction + +GitHub Actions is a powerful automation tool that enables CI/CD workflows directly within your GitHub repository. Securing your GitHub Actions workflows is crucial to protect your code, secrets, and infrastructure from potential security threats. + +This guide outlines best practices for securing your GitHub Actions workflows and minimizing security risks. + +## Table of Contents + +- [Secrets Management](#secrets-management) +- [Limiting Permissions](#limiting-permissions) +- [Third-Party Actions](#third-party-actions) +- [Dependency Management](#dependency-management) +- [Runner Security](#runner-security) +- [Pull Request Workflows](#pull-request-workflows) +- [OIDC Integration](#oidc-integration) +- [Audit and Monitoring](#audit-and-monitoring) + +## Secrets Management + +This section describes how secrets in GitHub Actions should be managed, teams should ensure that they are using robust secrets management tools such as Azure Key Vault and AWS Secrets Manager for securely storing secrets. + +### Use GitHub Secrets + +- Store sensitive data (API tokens, credentials, etc.) as [GitHub Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) +- Never hardcode sensitive values in your workflow files +- Do not use structured data as a secret - this can cause GitHubs secret redaction in logs to fail +- Rotate secrets regularly +- Use environment-specific secrets when possible +- Ensure a secret scanner is deployed as part of your workflows +- Public repositories should enable GitHub Secret Scanner and Push Protection + +### Minimize Secret Scope + +```yaml +# Good practice - limiting secret to specific environment +jobs: + deploy: + environment: production + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: Deploy + env: + API_TOKEN: ${{ secrets.API_TOKEN }} + run: ./deploy.sh +``` + +### Avoid Exposing Secrets in Logs + +- Don't echo or print secrets in workflow steps +- Set debug to false when using secrets +- Use masking for any dynamically generated secrets + +## Limiting Permissions + +### Use Least Privilege Principle + +Limit the GitHub token permissions to only what's necessary please [see here](https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication#permissions-for-the-github_token) for details on the default permissions that the github token is given when the permissions block is not used: + +```yaml +permissions: + contents: read + pull-requests: write + issues: write +``` + +### Use Fine-Grained Tokens + +Fine grained tokens *must* only be used if the GitHub token can not be used. + +- Create custom GitHub Apps with limited scopes when possible +- Use repository-scoped tokens instead of organization-wide tokens + +## Third-Party Actions + +While third-party actions can significantly enhance the functionality and efficiency of your workflows, they also introduce potential security risks: + +- *Untrusted Code*: Third-party actions are often maintained by external developers. If the code is not reviewed or vetted, it may contain vulnerabilities or malicious code that could compromise your repository or infrastructure. +- *Version Drift*: Using tags like @latest or branch references (e.g., @main) can lead to unexpected changes in behavior if the action is updated. This could introduce breaking changes or vulnerabilities into your workflows. +- *Dependency Vulnerabilities*: Third-party actions may rely on outdated or insecure dependencies, which could expose your workflows to known vulnerabilities. +- *Lack of Maintenance*: Some third-party actions may not be actively maintained, leaving them vulnerable to security issues or compatibility problems with newer GitHub Actions features. +- *Excessive Permissions*: Third-party actions may request more permissions than necessary, potentially exposing sensitive data or allowing unauthorized access to your repository. + +To mitigate these risks, always follow best practices, such as pinning actions to specific commit SHAs, reviewing the source code of actions, and using only trusted actions from reputable sources. + +### Pin Actions to Specific Versions + +When including a GitHub Action within your workflow you should perform due diligence checks to ensure that the action achieves the aims you are intending it to, and that it doesn't do anything unintended, this would include performing a code review of the GitHub action code. To prevent the underlying code being changed without your awareness always use specific commit SHAs instead of tags or branches as tags can be modified if the upstream repository is compromised: + +```yaml +# Not secure - can change unexpectedly +- uses: actions/checkout@v3 +# Better - using a specific version tag +- uses: actions/checkout@v3.1.0 +# Best - using a specific commit SHA +- uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b # v3.1.0 +``` + +### Verify Third-Party Actions + +When including a GitHub Action within your workflow consider alternatives, is there an existing mechanism you can use? Would this be something that could be reused and you could create your own action within the organisation that other teams could benefit from? If you can only achieve your goal with a third-party action then: + +- Only use trusted actions from the GitHub Marketplace +- Review the source code of third-party actions before using them +- Consider forking and maintaining your own copy of critical actions + +### Use Actions Security Best Practices + +- Enable Dependabot alerts for GitHub Actions +- Set up a workflow that regularly checks for outdated actions + +## Dependency Management + +### Scan Dependencies + +- Use dependency scanning tools like GitHub's Dependabot + +### Keep Dependencies Updated + +- Implement automated dependency updates +- Regularly review and update dependencies with security patches + +## Runner Security + +### Self-hosted Runner Security + +Self-hosted runners *must* only be [used with private repositories](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#self-hosted-runner-security). If using self-hosted runners: + +- Run them in isolated environments (containers/VMs) +- Regularly update and patch runner machines +- Implement proper network isolation +- Use ephemeral runners when possible + +```yaml +jobs: + build: + runs-on: [self-hosted, isolated] + steps: + # Your workflow steps here +``` + +### GitHub-hosted Runner Security + +- Be aware that GitHub-hosted runners are reset after each job +- Clean up any sensitive data before job completion +- Don't store persistent sensitive data in the runner's environment + +## Pull Request Workflows + +### Secure Pull Request Workflows + +- Don't expose secrets to pull request workflows from forks +- Use `pull_request_target` carefully with read-only permissions + +```yaml +# Safer approach for PR workflows +on: + pull_request: +jobs: + test: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@v3 + - name: Run tests + run: npm test +``` + +### Implement Required Reviews + +- Enforce branch protection rules +- Require code reviews before merging +- Use status checks to enforce security scans + +## OIDC Integration + +### Use OpenID Connect for Cloud Providers + +Instead of storing long-lived cloud credentials, use GitHub's OIDC provider: + +```yaml +jobs: + deploy: + runs-on: ubuntu-latest + permissions: + id-token: write + contents: read + steps: + - uses: actions/checkout@v3 + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions + aws-region: eu-west-2 +``` + +### Limit OIDC Token Claims + +- Set specific subject claims in your cloud provider +- Implement additional claim conditions (repository, branch, environment) + +## Audit and Monitoring + +### Enable Audit Logging + +- Monitor GitHub Actions usage via audit logs +- Set up alerts for suspicious activity + +### Review Workflow Changes + +- Enforce code reviews for workflow file changes +- Use CODEOWNERS to restrict who can modify workflow files + +```plaintext +# CODEOWNERS file/.github/workflows/ @security-team +``` + +### Regular Security Reviews + +- Conduct regular reviews of all workflows +- Update security practices based on emerging threats +- Monitor GitHub security advisories + +## Additional Resources + +- [GitHub Actions Security Hardening Guide](https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions) +- [GitHub Security Lab](https://securitylab.github.com/) +- [GitHub Actions Documentation](https://docs.github.com/en/actions) +- [Security for GitHub Actions](https://docs.github.com/en/actions/security-for-github-actions) + +## Conclusion + +Securing GitHub Actions requires a multi-layered approach focusing on secrets management, permissions, third-party action vetting, and proper configuration. By following these best practices, you can significantly reduce security risks while still enjoying the full benefits of GitHub Actions automation. + +Remember that security is an ongoing process - regularly review and update your security practices to adapt to new threats and challenges. diff --git a/practices/baseline-visibility.md b/practices/baseline-visibility.md new file mode 100644 index 00000000..4af702f0 --- /dev/null +++ b/practices/baseline-visibility.md @@ -0,0 +1,23 @@ +# Baseline Visibility + +## Collaborating + +This repository is part of the [NHSDigital GitHub Organisation](https://github.com/NHSDigital), which forms part of a wider [Enterprise](https://docs.github.com/en/enterprise-cloud@latest/admin/overview/about-github-for-enterprises). To enable effective collaboration between members of the NHSDigital organisation the [base permissions](https://docs.github.com/en/organizations/managing-user-access-to-your-organizations-repositories/managing-repository-roles/setting-base-permissions-for-an-organization) for repositories should be set to `No Permissions`, and visibility of Internal repositories is limited to members of the organisation. This will mean that individual members will not be able to view Private repository source code unless granted specific permissions. All members should be invited to the `everyone` team. This grants access to Internal repositories while keeping Private repository access constrained. + +As the organisation is part of the Enterprise there are other Organisations. Members of these other Organisations have read access to [Internal](./baseline-visibility.md#internal-repositories) repositories. + +## Contributing + +With a baseline of `No Permissions` on a repository and as a member of the `everyone` team, members of the Organisation can view the source code of Public and Internal repositories and Private repositories that they have explicit read permissions for. They might then be able to identify improvements to this code. Each repository should contain a [CONTRIBUTING.md](../CONTRIBUTING.md) to describe how both team members and members from the wider organisation can contribute to the repository. Contributions to repositories are welcomed and should be encouraged. + +## Code Owners + +Each repository should include a [CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) file in one of the [recommended locations](https://graphite.dev/guides/in-depth-guide-github-codeowners#creating-and-locating-your-codeowners-file). [Branch protections](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) must be in place to ensure that any Pull Requests are approved by users listed in the CODEOWNERS file. This ensures that any code changes are approved by appropriate members the teams that own and are responsible for the code and any downstream services that depend upon it. + +The presence of the CODEOWNERS file alows organisation members to identify who to contact should they have any questions about the code. + +Teams can be listed as CODEOWNERS, but secret teams should not be used. Otherwise potential contributors wouldn't be able to identify who the codeowners are. + +## Internal Repositories + +Non-public repositories can be created of the `Internal` type. All members of the Enterprise can view these repositories. This will include users who do not have direct access to the individual Organisation but who are members of the Enterprise. diff --git a/practices/cloud-databases.md b/practices/cloud-databases.md new file mode 100644 index 00000000..76b87cb5 --- /dev/null +++ b/practices/cloud-databases.md @@ -0,0 +1,22 @@ +# Cloud databases + +- [Cloud databases](#cloud-databases) + - [Context](#context) + - [Details](#details) + - [Infrastructure](#infrastructure) + +## Context + +- These notes are part of a broader set of [principles](../principles.md) +- Practices in this section contribute to [service reliability](service-reliability.md) +- See also [observability](observability.md) + +## Details + +### Infrastructure + +Successfully operating relational databases in a cloud environment, especially a serverless one, requires attention to the specific qualities of the database products as implemented by the cloud platforms. + +- Do use a small number of long-lived server instances. Do not create new servers per application environment. Database servers can be very slow to instantiate, which means that tools like `terraform` can time out waiting for them to start. Instead separate your infrastructure code between configuration for the application and configuration for the account, so that your servers are configured once per AWS account. The `terraform` code that configures an application instance should provision its resources by calling into the already-provisioned server in the account to create environment-specific databases. +- Do manage your database migrations. If you are using a web framework like Django, or Rails, you will have tooling built in to do this. Otherwise, use a tool like `alembic` if several people are likely to be working on features which change the database structure at the same time. If your needs are simpler, you may find a more straightforward approach - storing SQL scripts and sequentially running them, for instance - works just as well with fewer dependencies. If building a serverless application, deploy your migrations into a cloud function to be called by your `terraform` deployment. +- Learn how to refactor database schemas safely. Several coupled migrations and code changes may be needed to successfully change the schema with no downtime. diff --git a/practices/cloud-services.md b/practices/cloud-services.md index 97c2629d..fa74998f 100644 --- a/practices/cloud-services.md +++ b/practices/cloud-services.md @@ -16,7 +16,7 @@ ## Details -- Configure all infrastructure using declarative code such as Terraform and CloudFormation (see [everything as code](../patterns/everything-as-code.md)). +- Configure all infrastructure using declarative code such as Terraform or CloudFormation (see [everything as code](../patterns/everything-as-code.md)). - Automate monitoring and alerting (see [automate everything](../patterns/automate-everything.md) and [observability](observability.md). - Prefer serverless platform as a service (PaaS) over infrastructure as a service (IaaS) (see [outsource bottom up](../patterns/outsource-bottom-up.md)). - Where not serverless use ephemeral and immutable infrastructure. @@ -24,7 +24,7 @@ - Understand cloud supplier SLAs. - Make systems self-healing. - Prefer technologies which are resilient by default. - - Favour global-scoped (e.g. [CloudFront](https://aws.amazon.com/cloudfront/) or [Front Door](https://azure.microsoft.com/en-gb/pricing/details/frontdoor/)) or region-scoped services (e.g. [S3](https://aws.amazon.com/s3/), [Lambda](https://aws.amazon.com/lambda/), [Azure Functions](https://azure.microsoft.com/en-gb/products/functions/)) to availability-zone (AZ) scoped (e.g. [VMs](https://azure.microsoft.com/en-gb/services/virtual-machines/), [RDS DBs](https://aws.amazon.com/rds/)) or single-instance services (e.g. [EC2 instance storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)). + - Favour global-scoped (e.g. [CloudFront](https://aws.amazon.com/cloudfront/) or [Front Door](https://azure.microsoft.com/en-gb/pricing/details/frontdoor/)) or region-scoped services (e.g. [S3](https://aws.amazon.com/s3/), [Lambda](https://aws.amazon.com/lambda/), [Azure Functions](https://azure.microsoft.com/en-gb/products/functions/)) to availability-zone (AZ) scoped (e.g. [VMs](https://azure.microsoft.com/en-gb/products/virtual-machines/), [RDS DBs](https://aws.amazon.com/rds/)) or single-instance services (e.g. [EC2 instance storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)). - For AZ-scoped services, use redundancy to create required resilience (e.g. [AWS Auto Scaling Groups](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) or [Azure Scale/Availability Sets](https://docs.microsoft.com/en-us/azure/virtual-machines/availability)), and: - For stateless components use active-active configurations across AZs (e.g. running stateless containers across multiple AZs using [AWS Elastic Kubernetes Service](https://aws.amazon.com/eks/)) - For stateful components, e.g. databases, consider use of active-active configurations across AZs (e.g. [Aurora Multi-Master](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-multi-master.html)), but be aware of the added complexity conflict resolution for asynchronous replication can bring and potential performance impact where synchronous replication is chosen. @@ -36,14 +36,37 @@ - Services should scale automatically up and down. - If possible, drive scaling based on metrics which matter to users (e.g. response time), but balance this with the benefits of choosing leading indicators (e.g. CPU usage) to avoid slow scaling from impacting user experience. - Understand how rapidly demand can spike and ensure scaling can meet these requirements. Balance scaling needs with the desire to avoid over provisioning and use [pre-warming](https://petrutandrei.wordpress.com/2016/03/18/pre-warming-the-load-balancer-in-aws/) of judiciously where required. Discuss this with the cloud provider well before go live they can assist with pre-warming processes ([AWS](https://aws.amazon.com/premiumsupport/programs/iem/)). -- Infrastructure should always be fully utilised (if it isn't, it's generating waste). - - Though balance this with potential need to run with some overhead to accommodate failed instance replacement times without overloading remaining instances. +- As a rule of thumb, where you are using inelastic infrastructure, aim for 80% utilisation. + - Don't let utilisation rise far enough that a single instance failing would cause an outage. + - Too high utilisation will cause latency problems. Know what your performance SLOs are to understand how much latency headroom you have. - Keep up to date. - Services/components need prompt updates to dependencies where security vulnerabilities are found — even if they are not under active development. - Services which use deprecated or unsupported technologies should be migrated onto alternatives as a priority. - Understand and be able to justify vendor lock in (see [outsource from the bottom up](../patterns/outsource-bottom-up.md)). -- Build in [governance as a side effect](../patterns/governance-side-effect.md), e.g. - - Segregate production and non-production workloads. +- Build in [governance as a side effect](../patterns/governance-side-effect.md), e.g. + - Segregate production and non-production workloads + + Production and non-production workloads should be deployed into separate cloud subscriptions (Azure) or accounts (AWS) to enforce clear security boundaries, reduce risk of accidental impact and simplify policy enforcement. This separation enables: + + - Tighter access control for production, ensuring only the necessary users and automation have access + - Application of different Azure or AWS policies and guardrails (e.g. cost controls, logging requirements, monitoring sensitivity) + - Easier environment-specific cost tracking (especially in showback/chargeback models) + - Safer testing and change validation, supporting the DevOps approach of *"rapid, iterative and incremental change"* through controlled progression across environments (e.g. Dev → Int → NFT → Preprod → Prod) + + This structure is also aligned with the Cloud Adoption Framework for [Azure](https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/) and [AWS](https://aws.amazon.com/cloud-adoption-framework/), which recommend using subscriptions as units of governance and risk isolation. + + - Segregate products + + Each product should operate within its own set of cloud subscriptions (Azure) or accounts (AWS), rather than being co-located with other products in a large shared environment. This aligns infrastructure with product boundaries, enabling: + + - Empowered and autonomous teams (a core principle) to own, operate and iterate on their environments independently, enabling clear ownership and accountability + - Improved cost attribution for budgeting and forecasting, essential for long-living products supported by outcome teams + - Reduced risk of cross-product failure, misconfiguration or conflicting changes, and their blast radius + - Better alignment to [Conway’s Law](https://martinfowler.com/bliki/ConwaysLaw.html), [Team Topologies](https://teamtopologies.com/) and [Domain-Driven Design](https://martinfowler.com/bliki/DomainDrivenDesign.html), where infrastructure reflects the structure and ownership of the team, accelerating delivery and supporting flow + - Scalable approach to managing the product lifecycle: as each product evolves, is replatformed or retired, its resources can be managed in isolation + + By segregating subscriptions per product, we can reduce friction between teams, improve lifecycle management and support the [*"you build it, you run it"*](https://www.thoughtworks.com/en-gb/insights/decoder/y/you-build-it-you-run-it) DevOps approch. + -
Infrastructure must be tagged to identity the service so that unnecessary resources don't go unnoticed (click to expand) AWS Config rule to identify EC2 assets not tagged with "CostCenter" and "Owner": diff --git a/practices/feature-toggling.md b/practices/feature-toggling.md new file mode 100644 index 00000000..5d2b2306 --- /dev/null +++ b/practices/feature-toggling.md @@ -0,0 +1,172 @@ +# Feature Toggling + +- [Feature Toggling](#feature-toggling) + - [Context](#context) + - [The intent](#the-intent) + - [Key takeaway](#key-takeaway) + - [Background](#background) + - [What is feature toggling?](#what-is-feature-toggling) + - [Why use feature toggling?](#why-use-feature-toggling) + - [Types of toggles](#types-of-toggles) + - [Managing toggles](#managing-toggles) + - [Caveats](#caveats) + - [Toggling strategy](#toggling-strategy) + - [Toggle lifecycle](#toggle-lifecycle) + - [Best practice lifecycle](#best-practice-lifecycle) + - [Testing toggled features](#testing-toggled-features) + - [Designing for failure](#designing-for-failure) + - [Further reading](#further-reading) + +## Context + +- These notes are part of our broader [engineering principles](../principles.md). +- Feature toggling contributes to safer delivery, reduced deployment risk, and enhanced responsiveness to change. + +## The intent + +We use feature toggling as a **key enabling practice to support our move towards daily code integration and deployment, including multiple deployments per day** - this is the main motivation behind introducing this practice. Feature toggling allows us to separate deployment from release, so that incomplete features can be merged and deployed without impacting end users. It enables incremental development by allowing small, frequent commits to `main`, and supports risk-managed rollouts by enabling functionality selectively. Importantly, it also provides a mechanism for rapid rollback without reverting code. This approach is critical to mitigating the risks associated with frequent deployments, and is foundational to achieving a safe and sustainable continuous delivery model. + +## Key takeaway + +- Feature toggling enables functionality to be turned on or off without deploying new code. +- It separates deployment from release, allowing code to be safely deployed without activating a feature. +- Toggles should be explicitly managed with clear naming, documented intent, and timely removal. +- Toggle abuse (too many, long-lived, or undocumented flags) leads to tech debt and complex logic. + +## Background + +Feature toggling, also known as feature flags, is a technique for modifying system behaviour without changing code by checking a condition (usually externalised) at runtime. It is often used to control feature rollouts, manage risk, and test changes in production. + +This is particularly powerful in continuous delivery environments where small, frequent changes are the norm. It supports practices like canary releases, A/B testing, and operational kill switches. + +For a detailed and widely referenced introduction to this practice, see Martin Fowler's article on [Feature Toggles](https://martinfowler.com/articles/feature-toggles.html). + +While some areas are looking to adopt a more enterprise-grade offering with a dedicated feature toggling tool, it's important to recognise that more minimal feature toggle approaches may be appropriate for smaller or simpler systems. The [Thoughtworks Technology Radar](https://www.thoughtworks.com/radar) notes that many teams over-engineer feature flagging by immediately adopting complex platforms, when a simpler approach (e.g., environment variables or static config) would suffice. However, irrespective of how the toggle is implemented, the **governance, traceability, and lifecycle management processes should be consistent**. + +## What is feature toggling? + +Feature toggling works by introducing conditional logic into the application code. This logic evaluates a configuration value or remote toggle to determine whether to execute a new or existing code path. + +Toggles can be defined statically (e.g., environment variable or config file) or dynamically (e.g., via an external feature flag service). Dynamic toggles can be changed without restarting or redeploying the application. + +## Why use feature toggling? + +- **Decouple deployment from release**: Deploy code behind a toggle and activate it later. +- **Enable safe rollouts**: Enable features for specific users or teams to validate functionality before full rollout. +- **Support operational control**: Temporarily disable a feature which is causing issues, without needing to rollback. +- **Enable experimentation**: Run A/B tests to determine user impact. +- **Configure environment-specific behaviour**: Activate features in dev or test environments only. + +## Types of toggles + +According to Martin Fowler, toggles typically fall into the following categories: + +- **Release toggles**: Allow incomplete features to be merged and deployed. + +> [!NOTE] +> For teams practising daily integration and deployment, feature flagging is a foundational capability. It enables separation of deployment from release, allowing incomplete features to be merged, deployed, and safely hidden from users until ready. Where a product’s needs are focused on basic **canary releasing**, and the aspiration for daily deployment to production is yet to be realised, teams may choose to start with the native capabilities of their cloud provider (e.g., Azure deployment slots, AWS Lambda aliases, or traffic-routing rules). These offer infrastructure-level rollout control with minimal additional complexity or reliance on third-party tooling. + +- **Experiment toggles**: Support A/B or multivariate testing. + +> [!NOTE] +> True **A/B testing**, involving consistent user assignment, behavioural metrics, and statistical comparison, typically requires additional tooling to manage variant bucketing, exposure logging, and result analysis. In such cases, dedicated services are more appropriate, as the solutions can be subtle and mathematically complex (consider issues such as ["Optional Stopping" or "Peeking Bias"](https://www.evanmiller.org/how-not-to-run-an-ab-test.html)). We do not want to have to re-invent them. Teams are encouraged to start simple, and evolve their approach as feature granularity, targeting precision, analytical needs, and user expectations increase. + +- **Ops toggles**: Provide operational control for performance or reliability. +- **Permission toggles**: Enable features based on user roles or attributes. + +> [!WARNING] +> While permission toggles can target users by role or attribute during a rollout or experiment, they are not a replacement for robust, permanent role-based access control (RBAC). Use RBAC as a separate, first-class mechanism for managing user permissions. + +## Managing toggles + +Poorly managed toggles can lead to complexity, bugs, and technical debt. Best practices include: + +- Give toggles meaningful, consistent names. +- Document the purpose and expected lifetime of each toggle. +- Store toggle state in an observable system. +- Guard the feature behind a single toggle check, and pass the resulting behaviour or strategy through your code to minimise duplication and simplify removal. +- Ensure toggles are discoverable, testable, and auditable. +- Avoid nesting toggles or creating toggle spaghetti. +- Remove stale toggles once their purpose is fulfilled. Ideally integrate this into your CI pipeline to report on stale flags. + +Changing a toggle’s state can significantly impact system behaviour, especially in live environments. These changes should therefore be treated with the same rigour as code changes. Recommended practices include: + +- All toggle changes — whether in code or dynamic configuration stores — should be peer reviewed and documented. For code-based toggles, this naturally happens via pull requests. +- Use a feature toggle system or configuration store that logs all changes, including who made them, when, and the before/after values. +- Where possible, verify the impact of a toggle change in a non-production environment before enabling it in live. +- When enabling or disabling a toggle, communicate with relevant stakeholders, particularly for user-facing or operational changes. +- Where practical, manage toggle values through infrastructure-as-code or parameterised pipelines to allow version-controlled, reviewed, and auditable changes. + +## Caveats + +Whilst there are obvious benefits to Feature Toggling, there are some caveats worth bearing in mind when implementing them + +- **Performance Overhead**: Feature toggles can introduce performance overhead if they are checked frequently, especially within loops and every evaluation goes back to the server. +- **Toggle Bloat & Technical Debt**: Toggles are intended for short term use, failure to adhere to this principle can lead to conditional sprawl of if statements, harder code to read and maintain and increased risk of toggle conflicts or becoming orphaned +- **Test Complexity**: More toggles increase your permutations around a single test path. A single toggle doubles the test scenarios and needs careful factoring in to the test approach. +- **Increased Logging/Observability Needs**; Now need to know the state of the toggles at the point of the logs, otherwise inspecting the logs becomes incredibly difficult. + +## Toggling strategy + +Choose a feature flagging approach appropriate for the scale and complexity of your system: + +- **Simple applications**: Environment variables or configuration files. +- **Moderate scale and beyond**: Look to make use of a dedicated feature toggling tool, which supports targeting, analytics, and team workflows. + +Feature toggles should be queryable from all components that need access to their values. Depending on your architecture, this may require synchronisation, caching, or SDK integration. + +## Toggle lifecycle + +Toggles are intended to be short-lived unless explicitly designed to be permanent (e.g. permission toggles). + +### Best practice lifecycle + +1. **Introduce** the toggle with a clear purpose and target outcome. +2. **Keep it tidy** create a PR for the toggle removal called cleanup/feature_flag_name +3. **Implement** the feature behind the toggle. +4. **Test** the feature in both on/off states. +5. **Roll out** gradually (e.g., canary users, targeted groups). +6. **Monitor** the impact of the feature. +7. **Remove** the toggle once the feature is stable and fully deployed. + +Document toggles in your architecture or delivery tooling to ensure visibility and traceability. + +## Testing toggled features + +Features behind toggles should be tested in both their enabled and disabled states. This ensures correctness regardless of the toggle value. + +- Prefer testing the behaviour behind the toggle (e.g. via Strategy implementations) directly, rather than toggling features within tests. +- Where the Strategy Pattern is used, write separate unit tests for each strategy to validate their behaviour in isolation. +- If toggling is required in tests, use frameworks that allow injecting or mocking toggle values cleanly. +- Ensure integration and end-to-end tests include scenarios with the toggle both enabled and disabled, especially if the toggle is expected to persist across multiple releases. +- Include toggle state in test names or descriptions to clarify test intent (e.g. `shouldReturnNull_whenFeatureDisabled()`). +- Track test coverage across both toggle states and regularly review it for long-lived or critical toggles. +- Ensure test coverage includes edge cases introduced by toggled logic, such as different user roles, environment-specific behaviour, or state transitions. +- Use contract tests where toggled behaviour affects external APIs or integrations to ensure they remain backward-compatible. +- Avoid asserting on the presence or structure of toggle code itself, focus on testing expected outcomes. + +This is particularly important for toggles that persist for more than one release cycle. + +## Designing for failure + +If you are using a feature toggle service external to the application, feature toggles should never become a point of failure. Design your system so that it behaves predictably even if the toggle service is unavailable or fails to return a value. + +Best practices: + +- Default values: Every toggle should have a known and safe default (either on or off) hardcoded in the consuming service. +- Fail-safe logic: Ensure that remote flag checks have timeouts and fallback paths. +- Graceful degradation: Systems should still function, possibly with reduced capability, if a toggle cannot be resolved. +- Resilient integration: Ensure that SDKs or services used for toggling are resilient and do not block application startup or core functionality. + +## Further reading + +- [Feature Toggles (aka Feature Flags) by Martin Fowler](https://martinfowler.com/articles/feature-toggles.html) +- [Thoughtworks Tech Radar](https://www.thoughtworks.com/radar/techniques/simplest-possible-feature-toggle) +- [11 principles for building and scaling feature flag systems](https://docs.getunleash.io/topics/feature-flags/feature-flag-best-practices) +- [Best practices for coding with feature flags](https://launchdarkly.com/blog/best-practices-for-coding-with-feature-flags/) +- [Defensive coding](https://docs.flagsmith.com/guides-and-examples/defensive-coding) +- [An example tool for feature toggling](https://docs.flagsmith.com/) +- [How to use feature flags without technical debt](https://launchdarkly.com/blog/how-to-use-feature-flags-without-technical-debt/) +- [Scaling Feature Flags - A Roadmap for Safer Releases & Faster Development](https://143451822.fs1.hubspotusercontent-eu1.net/hubfs/143451822/eBooks/eBook:%20Scaling%20Feature%20Flags%20-%20A%20Roadmap%20for%20Safer%20Releases%20%26%20Faster%20Development.pdf) +- [Flip the Switch - Unlock Modern Software Development with Feature Flags](https://143451822.fs1.hubspotusercontent-eu1.net/hubfs/143451822/eBooks/Flip%20the%20Switch%20On%20Modern%20Software%20Development%20with%20Feature%20Flags%20-%20Flagsmith.pdf) +- [Modern Development Practices in Banking: A Playbook](https://143451822.fs1.hubspotusercontent-eu1.net/hubfs/143451822/eBooks/Modern%20Development%20Practices%20in%20Banking%20by%20Flagsmith.pdf) diff --git a/practices/guides/commit-purge.md b/practices/guides/commit-purge.md index fea01f86..5562cae6 100644 --- a/practices/guides/commit-purge.md +++ b/practices/guides/commit-purge.md @@ -2,11 +2,21 @@ ## Overview -Engineering teams should take all necessary precautions to ensure that sensitive data does not leak into Source Control Management Systems. This includes secrets being pushed to a remote branch, as well as merging into the default branch. Teams should consider **any** secret posted to a branch of a public repository as compromised and should take necessary steps to revoke and rotate this secret. For Private and Internal repositories teams should still treat leaked credentials as compromised and revoke and rotate them. Teams should also review their Near Miss reporting requirements and ensure that necessary steps are taken. Teams should ensure that a [Secret scanner](https://github.com/NHSDigital/software-engineering-quality-framework/tree/main/tools/nhsd-git-secrets) is enabled on their repositories. Teams must also ensure that developers follow standard processes to ensure any pre-commit hooks are enabled and enforced to reduce the risk of sensitive information being accidentally published. Teams should also contribute to the rule set for these tools as appropriate to ensure secrets are identified correctly. +There are multiple steps required to ensure sensitive data committed to a GitHub hosted Git repository is fully removed. + +Engineering teams must take all necessary precautions to ensure that sensitive data does not leak into Source Control Management Systems. This includes secrets being pushed to a remote branch, as well as merging into the default branch. Teams must consider **any** secret posted to a branch of a public repository as compromised and must take necessary steps to revoke and rotate this secret. For Private and Internal repositories teams must still treat leaked credentials as compromised and revoke and rotate them. + +Teams must also review their Near Miss reporting requirements and ensure that necessary steps are taken. + +Teams must ensure that a [Secret scanner](https://github.com/NHSDigital/software-engineering-quality-framework/tree/main/tools/nhsd-git-secrets) is enabled on their repositories. + +Teams must also ensure that developers follow standard processes to ensure any pre-commit hooks are enabled and enforced to reduce the risk of sensitive information being accidentally published. Teams should also contribute to the rule set for these tools to ensure secrets are identified correctly. If a secret or other sensitive information is identified as having been pushed to a remote repository in GitHub then the following steps ***must*** be undertaken to ensure removal of the information. Please note that just removing the data from the git history is **not** sufficient as views can be cached by the UI. -1. Rotate the secrets that have been revealed – whether the repository is public or private this should be a key step in reducing the risk of any accidental publishing of secrets. -2. Consider whether an incident should be raised, for example has sensitive information been shared in a public repository. If in doubt raise an incident following your internal processes. +### Remediation Steps + +1. Rotate the secrets that have been revealed – whether the repository is public or private this is a key step in reducing the risk of any accidental publishing of secrets. +2. A security incident **must** be raised for all sensitive date committed. This ensures that our Cyber teams can support in assessing the level of risk of the exposure. Contributors must raise an incident following your internal processes. 3. Undertake steps to [remove the sensitive data from your Git history](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository#purging-a-file-from-your-repositorys-history). 4. Once the history has been cleansed we need to request that GitHub purge their cache – please [raise a request](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository#fully-removing-the-data-from-github) with the internal Github admins mailbox. diff --git a/practices/guides/commit-signing.md b/practices/guides/commit-signing.md index 0c4d3eff..70311425 100644 --- a/practices/guides/commit-signing.md +++ b/practices/guides/commit-signing.md @@ -1,90 +1,203 @@ # Git commit signing setup guide -- [Git commit signing setup guide](#git-commit-signing-setup-guide) - - [From Workstations](#from-workstations) - - [macOS](#macos) - - [Windows](#windows) - - [From Pipelines](#from-pipelines) - - [GitHub Actions](#github-actions) - - [AWS CodePipeline](#aws-codepipeline) - - [Troubleshooting](#troubleshooting) +Using GPG, SSH, or S/MIME, you can sign commits and tags locally. These commits and tags are marked as verified on GitHub so other people can be confident that the changes come from a trusted source (see the full GitHub documentation [here](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)). -## From Workstations +> You should only set up **one** of these options - **don't attempt to set up GPG and SSH commit signing**! -### macOS +The instructions on this page focus on the recommended method - GPG. -- Install the [Brew package manager](https://brew.sh) +## GPG commit signing -```bash -brew upgrade -brew install gnupg pinentry-mac -gpg --full-generate-key -``` +### From Workstations -- Accept the defaults, Curve 25519 etc. -- Enter your GitHub account name as the Real Name -- Enter your GitHub account email as the Email Address -- You can use the privacy *@users.noreply.github.com* email address listed in the GitHub profile: *Settings > Email* -- Define a passphrase for the key and keep it in your password manager +If you have already committed and need to retrospectively sign commits, follow the instructions below, then follow the [retrospective commit signing instructions](./retrospective-commit-signing.md). -```bash -gpg --armor --export ${my_email_address} | pbcopy -``` +#### macOS -- Public key is now in your clipboard - in your GitHub account add it to your profile via *Settings > SSH and GPG Keys> Add New GPG Key* -- Paste it in +1. Install `gnupg` & `pinentry-mac` with [Brew](https://brew.sh): -```bash -git config --global user.email ${my_email_address} # same one used during key generation -git config --global user.name ${my_username} -git config --global commit.gpgsign true -sed -i '' '/^export GPG_TTY/d' ~/.zshrc -echo export GPG_TTY=\$\(tty\) >> ~/.zshrc -source ~/.zshrc -PINENTRY_BIN=$(whereis -q pinentry-mac) -sed -i '' '/^pinentry-program/d' ~/.gnupg/gpg-agent.conf -echo "pinentry-program ${PINENTRY_BIN}" >> ~/.gnupg/gpg-agent.conf -gpgconf --kill gpg-agent -``` + ```bash + brew upgrade + brew install gnupg pinentry-mac + sed -i '' '/^export GPG_TTY/d' ~/.zshrc + echo export GPG_TTY=\$\(tty\) >> ~/.zshrc + source ~/.zshrc + PINENTRY_BIN=$(whereis -q pinentry-mac) + mkdir -p ~/.gnupg + touch ~/.gnupg/gpg-agent.conf + sed -i '' '/^pinentry-program/d' ~/.gnupg/gpg-agent.conf + echo "pinentry-program ${PINENTRY_BIN}" >> ~/.gnupg/gpg-agent.conf + gpgconf --kill gpg-agent + ``` -The first time you commit you will be prompted to add the GPG key passphrase to the macOS Keychain. Thereafter signing will happen seamlessly without prompts. +1. Create a new GPG key: -Most of the published solutions for this don't work because *brew* seems to have moved the default folder for binaries, plus many guides contain obsolete settings for *gpg-agent*. + ```bash + gpg --full-generate-key + ``` -### Windows + 1. Pick `(9) ECC (sign and encrypt)` then `Curve 25519` ([Ed25519](https://en.wikipedia.org/wiki/EdDSA#Ed25519) offers the strongest encryption at time of writing) + 1. Select a key expiry time (personal choice) + 1. `Real name` = Your GitHub handle + 1. `Email address` = An email address [registered against your GitHub account](https://github.com/settings/emails) - to enable [Smart Commits](https://nhsd-confluence.digital.nhs.uk/x/SZNYRg#UsingtheGitHubintegrationinJira-SmartCommits) ([Jira/GitHub integration](https://support.atlassian.com/jira-software-cloud/docs/process-issues-with-smart-commits/)), use your `@nhs.net` address -- Install [Git for Windows](https://git-scm.com/download/win), which includes Bash and GnuPG -- Right-click on the Desktop > *Git Bash Here* + > If instead you opt for the private *@users.noreply.github.com* email address, consider enabling `Block command line pushes that expose my email`. -```bash -gpg --full-generate-key -``` + 1. Avoid adding a comment (this *may* prevent git from auto-selecting a key - see Troubleshooting section below) + 1. Review your inputs and press enter `O` to confirm + 1. Define a passphrase for the key -- Pick *RSA and RSA*, or *RSA (sign only)* - there is no elliptic curve cryptography (ECC) support at the time of writing -- Set key size to 4096 bit, the minimum accepted for GitHub -- Enter your GitHub account name as the Real Name -- Enter your GitHub account email as the Email Address -- You can use the privacy *@users.noreply.github.com* email address listed in the GitHub profile: *Settings > Email* -- Define a passphrase for the key and keep it in your password manager +1. Test the key is visible and export the PGP public key (to your clipboard): -```bash -gpg --armor --export ${my_email_address} | clip -``` + ```bash + gpg -k # This should list the new key + gpg --armor --export | pbcopy + ``` -- Public key is now in your clipboard - in your GitHub account add it to your profile via *Settings > SSH and GPG Keys> Add New GPG Key* -- Paste it in + > Your PGP public key is now in your clipboard! -```bash -git config --global user.email ${my_email_address} # same one used during key generation -git config --global user.name ${my_username} -git config --global commit.gpgsign true -``` +1. [Add the public key to your GitHub account](https://github.com/settings/gpg/new) (`Settings` -> `SSH and GPG keys` -> `New GPG key`) + + > Note the `Key ID` as you'll need this in the next step. + +1. Set your local git config to use GPG signing: + + ```bash + git config --global user.email # same one used during key generation + git config --global user.name + git config --global user.signingkey + git config --global commit.gpgsign true + git config --global tag.gpgsign true + ``` + +1. Test it works: + + 1. Create a temporary branch of your favourite repository. + 1. Make an inconsequential whitespace change. + 1. Commit the change. + 1. You will be prompted for your GPG key passphrase - optionally select to add it to the macOS Keychain. + 1. Check the latest commit shows a successful signing: + + ```bash + $ git log --show-signature -1 + ... + gpg: Good signature from " <>" [ultimate] + Author: <> + ... + ``` + +#### Windows/WSL + +1. Install (as administrator) [Git for Windows](https://git-scm.com/download/win) (which includes Bash and GnuPG) +1. Open `Git Bash` +1. Create a new GPG key: + + ```bash + gpg --full-generate-key + ``` + + 1. Pick `(9) ECC (sign and encrypt)` then `Curve 25519` ([Ed25519](https://en.wikipedia.org/wiki/EdDSA#Ed25519) offers the strongest encryption at time of writing) + + > If you already had Git for Windows installed, and its version is between `2.5.0` and `2.30.x`, the `(9) ECC and ECC` option is available if you run `gpg --expert --full-generate-key`, however if you can upgrade to the latest version, this is advised. + + 1. Select a key expiry time (personal choice) + 1. `Real name` = Your GitHub handle + 1. `Email address` = An email address [registered against your GitHub account](https://github.com/settings/emails) - to enable [Smart Commits](https://nhsd-confluence.digital.nhs.uk/x/SZNYRg#UsingtheGitHubintegrationinJira-SmartCommits) ([Jira/GitHub integration](https://support.atlassian.com/jira-software-cloud/docs/process-issues-with-smart-commits/)), use your `@nhs.net` address + + > If instead you opt for the private *@users.noreply.github.com* email address, consider enabling `Block command line pushes that expose my email`. + + 1. Avoid adding a comment (this *may* prevent git from auto-selecting a key - see Troubleshooting section below) + 1. Review your inputs and press enter `O` to confirm + 1. A new window called pinentry will appear prompting you to enter a passphrase. + +1. Test the key is visible and export the PGP public key (to your clipboard): + + ```bash + gpg -k # This should list the new key + gpg --armor --export | clip + ``` + + > Your PGP public key is now in your clipboard! + +1. [Add the public key to your GitHub account](https://github.com/settings/gpg/new) (`Settings` -> `SSH and GPG keys` -> `New GPG key`) + + > Note the `Key ID` as you'll need this in the next step. + +1. Set your local git config to use GPG signing: + + ```bash + git config --global user.email # same one used during key generation + git config --global user.name + git config --global user.signingkey + git config --global commit.gpgsign true + git config --global tag.gpgsign true + ``` + +1. Now your key is created, make it available within Windows: + + 1. Export the key: + + ```bash + gpg --output .pgp --export-secret-key + ``` + + 1. Install (as administrator) [Gpg4win](https://www.gpg4win.org/) (which includes GnuPG and Kleopatra) + + > **Ensure both `GnuPG` and `Kleopatra` are installed!** + + 1. Open Kleopatra -> `Import` -> Select the `.pgp` file created in the first step + 1. In `cmd`, test the key is visible and set your local git config to use GPG signing: + + ```bash + gpg -k # This should list the new key + git config --global user.email # same one used during key generation + git config --global user.name + git config --global user.signingkey + git config --global commit.gpgsign true + git config --global tag.gpgsign true + ``` + +1. Now make it available within WSL: + + 1. Within Ubuntu: + + ```bash + sudo ln -s /mnt/c/Program\ Files\ \(x86\)/GnuPG/bin/gpg.exe /usr/local/bin/gpg + sudo ln -s gpg /usr/local/bin/gpg2 + ``` + + 1. Close and reopen your Ubuntu terminal + + 1. Test the key is visible and set your local git config to use GPG signing: + + ```bash + gpg -k # This should list the new key + git config --global user.email # same one used during key generation + git config --global user.name + git config --global user.signingkey + git config --global commit.gpgsign true + git config --global tag.gpgsign true + ``` + +1. Test it works: + + 1. Create a temporary branch of your favourite repository. + 1. Make an inconsequential whitespace change. + 1. Commit the change. + 1. You will be prompted for your GPG key passphrase. + 1. Check the latest commit shows a successful signing: -When you commit you will be prompted to enter the GPG key passphrase into a Pinentry window. + ```bash + $ git log --show-signature -1 + ... + gpg: Good signature from " <>" [ultimate] + Author: <> + ... + ``` -## From Pipelines +### From Pipelines -### GitHub Actions +#### GitHub Actions A GitHub Actions workflow will by default authenticate using a [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication) which is generated automatically. @@ -95,7 +208,7 @@ The workflow would then use a Personal Access Token, stored with the GPG private ```yaml steps: - name: Checkout - uses: actions/checkout@v3 + uses: actions/checkout@v5 with: token: ${{ secrets.BOT_PAT }} ref: main @@ -119,7 +232,7 @@ git commit ${GITHUB_SIGNING_OPTION} -am "Automated commit from GitHub Actions: $ git push ``` -### AWS CodePipeline +#### AWS CodePipeline The cryptographic libraries in the default Amazon Linux 2 distro are very old, and do not support elliptic curve cryptography. When using pre-existing solution elements updating the build container is not always an option. This restricts the GPG key algorithm to RSA. You should use RSA-4096, which is the required minimum for GitHub. @@ -136,7 +249,7 @@ if [[ ${BOT_SSH_KEY} != "None" ]]; then echo "StrictHostKeyChecking yes" >> ~/.ssh/config echo "UserKnownHostsFile=~/.ssh/known_hosts" >> ~/.ssh/config echo "${BOT_SSH_KEY}" > ~/.ssh/ssh_key - echo -e "\n\n" >> ~/.ssh/ssh_key + echo -e "\n\n" >> ~/.ssh/ssh_key chmod 600 ~/.ssh/ssh_key eval "$(ssh-agent -s)" ssh-add ~/.ssh/ssh_key @@ -172,8 +285,8 @@ git commit ${GITHUB_SIGNING_OPTION} -am "Automated commit from ${SCRIPT_URL}" git push ``` -## Troubleshooting +### Troubleshooting -Re-run your git command prefixed with GIT_TRACE=1 +Re-run your git command prefixed with `GIT_TRACE=1`. -A failure to sign a commit is usually because the name or email does not quite match those which were used to generate the GPG key, so git cannot auto-select a key. Ensure that these are indeed consistent. You are able to [force a choice of signing key](https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key), though this should not be necessary. +A failure to sign a commit is usually because the name or email does not quite match those which were used to generate the GPG key, so git cannot auto-select a key. Ensure that these are indeed consistent. (If you added a comment when creating your GPG key, this *may* cause a mismatch: the comment will be visible when listing your GPG keys, e.g. `RealName (Comment) `.) You are able to [force a choice of signing key](https://docs.github.com/en/authentication/managing-commit-signature-verification/telling-git-about-your-signing-key), though this should not be necessary. diff --git a/practices/guides/retrospective-commit-signing.md b/practices/guides/retrospective-commit-signing.md new file mode 100644 index 00000000..b29141fa --- /dev/null +++ b/practices/guides/retrospective-commit-signing.md @@ -0,0 +1,77 @@ +# Retrospective commit signing guide + +- [Retrospective commit signing guide](#retrospective-commit-signing-guide) + - [Signing previously pushed commits](#signing-previously-pushed-commits) + - [Steps](#steps) + +## Signing previously pushed commits + +If you have pushed a commit without signing this can result in your PR not being able to be merged into the main or default branch. The following steps will guide you through retrospectively signing your commits. Note you can sign multiple commits if required. + +Please take the time to understand the commands that you are using, this is just a guide. + +### Steps + +1. **Identify unsigned commits** + You have a branch that contains one or more unsigned commits. In the screenshot below, there are two unsigned commits followed by two commits showing the `Verified` label, which indicates they were signed. + + ![Commit history showing unsigned and signed commits](../../images/commit-history-github.png) + +2. **Understand the issue** + The first two commits aren't verified, and therefore the merge to the `main` branch is not allowed: + + ![Merging is blocked as the commits aren't signed](../../images/merging-is-blocked-signatures.png) + +3. **Switch to the branch with unsigned commits** + Go to your CLI and ensure that you are on the branch with the unsigned commits. + +4. **Start an interactive rebase** + Issue the following command: + + ```bash + git rebase -i --root + ``` + + This puts the editor into interactive mode for rebase. You will see the commit history as shown in the screenshot below: + + ![Interactive rebase before selecting commits](../../images/interactive-rebase-before-selecting-commits.png) + +5. **Mark commits for editing** + Scroll down the list until you find the commits you want to sign. Change the keyword `pick` to `edit` for those commits. + + ![Interactive rebase after selecting commits](../../images/interactive-rebase-after-selecting-commits.png) + + If you are using `Nano`, save the changes with `Ctrl+X` and confirm with `Enter`. For `Vi`, exit with `:wq` to save and quit. + +6. **Amend the commit to include a signature** + For each commit you flagged as `edit`, run the following commands: + + ```bash + git commit -S --amend --no-edit + git rebase --continue + ``` + + Rebase will cycle through the commits you flagged for editing: + + ![Cycling through commits flagged for editing](../../images/cycle-through-flagged-commits.png) + + Repeat the `amend` and `continue` steps for each commit. + +7. **Complete the rebase** + Once rebasing is complete, you will see a message like: + + ```plaintext + Successfully rebased and updated refs/heads/… + ``` + +8. **Push the changes** + Push the updated commits back to your branch. Use a force push if necessary: + + ```bash + git push -f + ``` + +9. **Verify the changes** + Refresh the browser window for your PR. You should now see the verified commits: + + ![Updated commit history in GitHub](../../images/updated-commit-history-github.png) diff --git a/practices/open-source.md b/practices/open-source.md new file mode 100644 index 00000000..81df3f43 --- /dev/null +++ b/practices/open-source.md @@ -0,0 +1,199 @@ +# Open Source + +- [Open Source](#open-source) + - [1. Context](#1-context) + - [2. Open sourcing our own code](#2-open-sourcing-our-own-code) + - [3. Open source dependencies](#3-open-source-dependencies) + - [3.1 If your application is an API-only service](#31-if-your-application-is-an-api-only-service) + - [3.1.1 If your application is coded in the open, or is to be open source](#311-if-your-application-is-coded-in-the-open-or-is-to-be-open-source) + - [3.1.1.1 GPL-2.0](#3111-gpl-20) + - [3.1.1.2 GPL-3.0](#3112-gpl-30) + - [3.1.1.3 LGPL-3.0](#3113-lgpl-30) + - [3.1.1.4 AGPL-3.0](#3114-agpl-30) + - [3.1.2 If your application is closed source](#312-if-your-application-is-closed-source) + - [3.2 If someone else runs your application](#32-if-someone-else-runs-your-application) + - [3.2.1 GPL-2.0](#321-gpl-20) + - [3.2.2 GPL-3.0](#322-gpl-30) + - [3.2.3 LGPL-3.0](#323-lgpl-30) + - [3.3 If your application runs in the user's browser](#33-if-your-application-runs-in-the-users-browser) + - [3.3.1 If your application is coded in the open, or is to be open source](#331-if-your-application-is-coded-in-the-open-or-is-to-be-open-source) + - [3.3.1.1 If your server and client components are distributed together](#3311-if-your-server-and-client-components-are-distributed-together) + - [3.3.1.2 If your server and client components are distributed separately](#3312-if-your-server-and-client-components-are-distributed-separately) + - [3.3.2 If your application is closed source](#332-if-your-application-is-closed-source) + - [4. The LGPL: on Linking](#4-the-lgpl-on-linking) + - [4.1 JavaScript on the web](#41-javascript-on-the-web) + - [4.2 JavaScript on the server (node.js/bun)](#42-javascript-on-the-server-nodejsbun) + - [5. What to do when a dependency changes licence](#5-what-to-do-when-a-dependency-changes-licence) + - [5.1 Comply with the new licence](#51-comply-with-the-new-licence) + - [5.2 Stay on the old version](#52-stay-on-the-old-version) + - [5.3 Replace the dependency](#53-replace-the-dependency) + +This section provides guidance for teams who either consume open source dependencies, or want to understand our responsibilities when we [code in the open](https://service-manual.nhs.uk/standards-and-technology/service-standard-points/12-make-new-source-code-open). + +## 1. Context + +- These notes are part of a broader set of [principles](../principles.md). +- They represent the Engineering community's interpretation of legal advice and licensing best practice for the context of software application delivery within and by NHS England, and should not be read as definitive outside that context. + +## 2. Open sourcing our own code + +[Item 12 in the Gov.UK Service Standard](https://www.gov.uk/service-manual/service-standard/point-12-make-new-source-code-open) reads as follows: + +> Make all new source code open and reusable, and publish it under appropriate licences. Or if this is not possible, provide a convincing explanation of why this cannot be done for specific subsets of the source code. + +We inherit this rule into NHS England's projects via the [NHS Service Manual](https://service-manual.nhs.uk/standards-and-technology/service-standard-points/12-make-new-source-code-open). + +There is further guidance as to when publishing our source code would not be appropriate in the (draft) [Open Source Policy](https://github.com/nhsx/open-source-policy/blob/main/open-source-policy.md#3-when-code-should-be-open-or-closed). + +The position of all three of these documents is that we should code in the open by default. If you are starting a new project then your first choice should be the [MIT Licence](https://choosealicense.com/licenses/mit/). See [here](https://github.com/nhsx/open-source-policy/blob/main/open-source-policy.md#c-licences-and-regulatory-requirements) for the rationale, and for when alternatives may be appropriate. + +There will be special cases where we need to deviate from the licences listed in the above guidance. For instance, if we were to build a reference implementation of an API standard, and we wanted to ensure that future developments of that reference implementation benefit the public commons, it may be suitable to choose the AGPL for that implementation. In such cases, where you believe that you may need to deviate from the [Open Source policy](https://github.com/nhsx/open-source-policy/blob/main/open-source-policy.md), you should consult the Engineering Technical Authority for advice. While it is not (yet) a binding policy, it is a distillation of good practice and should be followed unless there is a good reason not to, as in this example. + +## 3. Open source dependencies + +All of our projects rely on open source dependencies to some degree. We need to have confidence that we can comply, and are complying, with the terms of the licences which grant us permission to use those dependencies. Actions that we need to take will depend on how the application you are building will be deployed. + +We must not use a dependency which does not specify a licence. A dependency with no licence grants us no rights to distribute what we build. + +Further, we must constrain our dependencies to those with licences that we understand. There are a lot of licences out there, and we can't possibly look at all of them. The majority of candidate dependencies will use one of a small number of licences. If there is a dependency that you need to use which is only available under a licence that we do not currently list, consult the Engineering Technical Authority. The Engineering Board may approve a one-off use; or we may choose to update this guidance to advise that the licence in question is suitable for general use. + +Dependencies with the following licences are unconditionally acceptable: + +- [AFL-3.0](https://opensource.org/license/afl-3-0-php) +- [Apache-2.0](https://opensource.org/license/apache-2-0) +- [BSL-1.0](https://opensource.org/license/BSL-1.0) +- [BSD-2-Clause](https://opensource.org/license/BSD-2-Clause) +- [BSD-3-Clause](https://opensource.org/license/BSD-3-Clause) +- [ECL-2.0](https://opensource.org/license/ecl-2-0) +- [MIT](https://opensource.org/license/MIT) + +All of these licences require that the authors of the dependencies are credited when our applications are distributed. + +While it is not intended as a software licence, it is not uncommon to encounter dependencies with the [CC0-1.0](https://creativecommons.org/public-domain/cc0/) "No Rights Reserved" licence. This is also unconditionally acceptable. + +Further rules apply, and dependencies with other licence types may be used, depending on the specifics of your project. + +### 3.1 If your application is an API-only service + +In this case, your application is never distributed to anyone else in either source code or executable form. + +#### 3.1.1 If your application is coded in the open, or is to be open source + +If your application is being developed in the open, or to be open sourced at release, then you may use dependencies with the following licences under additional constraints: + +##### 3.1.1.1 [GPL-2.0](https://opensource.org/license/GPL-2.0) + +Your application's source code must be available under the GPL, version 2.0 or later. + +##### 3.1.1.2 [GPL-3.0](https://opensource.org/license/GPL-3.0) + +Your application's source code must be available under the GPL version 3.0. + +##### 3.1.1.3 [LGPL-3.0](https://opensource.org/license/LGPL-3.0) + +If your application only uses the interface of the dependency (as would commonly be the case), then there is no obligation. + +If your application directly copies the source code of the dependency, then your application's source code must be available under either the GPL version 3.0 or the LGPL version 3.0. + +##### 3.1.1.4 [AGPL-3.0](https://opensource.org/license/agpl-v3) + +The AGPL is designed to ensure that the source code is available for services made available to the user over a network. We may decide that this is suitable for some of our services: see above. + +If your application's source code is already intended to be made available under the AGPL version 3.0, then you may use a dependency with this licence. + +#### 3.1.2 If your application is closed source + +You may use dependencies licensed under these licences without constraint: + +- [GPL-2.0](https://opensource.org/license/GPL-2.0) +- [GPL-3.0](https://opensource.org/license/GPL-3.0) +- [LGPL-3.0](https://opensource.org/license/LGPL-3.0) + +You may not use dependencies licensed under the AGPL-3.0. There may be particular circumstances where legally this may be possible, but it is not practical for us to engage with the specifics. In this case you should contact the Engineering Technical Authority with a view to obtaining the dependency under a different licence, or identifying a replacement. + +### 3.2 If someone else runs your application + +In these cases, you are distributing your application to the user at least in executable form, and possibly in source code form too. This distribution means that you may use dependencies with the following licences under additional constraints, _whether or not your application is open or closed source_. If the application you are developing must be closed source, you must either make it open source or refrain from dependencies with these licences: + +#### 3.2.1 [GPL-2.0](https://opensource.org/license/GPL-2.0) + +Your application's source code must be available under the GPL, version 2.0 or later. + +#### 3.2.2 [GPL-3.0](https://opensource.org/license/GPL-3.0) + +Your application's source code must be available under the GPL version 3.0. + +#### 3.2.3 [LGPL-3.0](https://opensource.org/license/LGPL-3.0) + +If your application only uses the interface of the dependency (as would commonly be the case), then there is no obligation. + +If your application directly copies the source code of the dependency, then your application's source code must be available under either the GPL version 3.0 or the LGPL version 3.0. + +### 3.3 If your application runs in the user's browser + +This case is a mix of the previous two: part of the application is a (number, possibly zero, of) API service(s) running on infrastructure you own or control; another part is a (number, at least one, of) user interfaces delivered to the user's browser and running on their computer. + +#### 3.3.1 If your application is coded in the open, or is to be open source + +##### 3.3.1.1 If your server and client components are distributed together + +If you develop both parts together, where the source code for the API component(s) are distributed with the source code for the user interface, you should apply the same licence to both parts. Multiple licences in the same source code distribution is legally possible but too complex for our purposes. If you find yourself needing to do this, you should split the distribution into separately-licenced parts. + +Where the server and client components are distributed together, you must satisfy licence constraints implied by the dependencies of both. Delivering a client to be executed in the user's browser is legally equivalent to delivering a desktop application, and must follow the same rules: attribution notices must be retained made available as specified in the licences of the dependencies bundled in the client, and the terms of any copyleft licenses must be observed. + +The effect of the GPL (version 2.0 or 3.0) is notable here. A dependency licensed under the GPL that is only used on the server would ordinarily not affect decisions as to which licence to use for the client, but because of the above single-licence rule for where components are distributed together, that is no longer true here. A dependency licensed under the GPL on the server here requires the whole distribution, including the client, to be made available under the GPL. + +##### 3.3.1.2 If your server and client components are distributed separately + +Separate distributions enable you to use different licences for the server and the client components, and may incur different responsibilities. For instance, if you are developing a node.js/React application with a GPL'd dependency which is only used on the server, and which is excluded from the client that the user downloads, then the requirements of the GPL do not apply to the client. + +Follow the rules above for an API-only service for the server components, and follow the rules for a desktop app above for the client components. + +#### 3.3.2 If your application is closed source + +A closed source web application will, with one exceptional licence, only have to be concerned with the dependencies bundled and distributed with the client, for which the rules above in section 3.2 apply. + +That exceptional licence is the AGPL. Dependencies licensed under the AGPL should never be used in closed source applications. + +## 4. The LGPL: on Linking + +The LGPL was created to ensure that a library could be used without the software incorporating it accruing any licensing obligations, while retaining the property from the GPL that the user be able to modify it and substitute their own version. It expresses this with reference to "object code", "linking", and "headers", which are technical terms relating to the specific details of the C and C++ languages commonly in use among the Free Software communities at the time the LGPL was drafted. These terms do not have the same meanings in our current preferred development environments, but we can interpret the intent of the licence to arrive at a practical policy through which to clarify our obligations under specific scenarios. + +### 4.1 JavaScript on the web + +It is our policy that: + +- Loading an LGPL'd library unmodified via a `