Entra ID Testing With Maester
Earlier this month, I came across a tool called Maester. It is a Microsoft security test tool. It can be used to test security in Entra ID and M365 (Exchange and SharePoint). It has tests for CIS, CISA, EIDSCA, its own Maester tests, and it is possible to create your own custom tests. Maester is an open source project, and the core team members are Fabian Bader, Thomas Naunheim, Mike Soule, and Merill Fernando. Everybody can contribute on github.
Setup/Configuration
On Maester, they have multiple methods that can be used to set up the tool. I choose to use the Azure DevOps pipeline. The guide for how to set it up is quite easy to follow. But I chose to make some changes, and therefore, I got some challenges that I needed to solve. So the steps are like this:
- Create an Azure DevOps organisation (if you don’t have one), and create a new project.
- Import the measter-test repository to the new project.
- Create an identity federation between Azure DevOps and your Azure tenant.
- Give the app registration the api permissions it needs.
- Create a pipeline in the repository you imported.
- And here my problems started
Modifications
During the creation of the pipeline, i did choose to use the Virtual Machine Scale Set (VMSS) hosted agent. I did this because I have my own runner in my homelab. So the first change I did was to point the pipeline to the correct runner:
pool:
name: 'homelab'
When running the pipeline after this, I get this error:
The solution for this was that in the pipeline on line 60, there is an import of import-module /usr/share/microsoft.graph_2.19.0/Microsoft.Graph.Authentication. Maester is importing another version of this module during the import of the Maester module, so it creates a mismatch in versions. So the solution I went for was to comment out that line, and then the pipeline will work. Before I ran the pipeline, I changed which tests to run. I wanted all tests to run. To do this, I changed the following line in the pipeline:
Invoke-Maester -Path $(System.DefaultWorkingDirectory)/tests/Maester/ -PesterConfiguration $PesterConfiguration -OutputFolder '$(System.DefaultWorkingDirectory)/test-results'
To:
Invoke-Maester -Path $(System.DefaultWorkingDirectory)/tests/ -PesterConfiguration $PesterConfiguration -OutputFolder '$(System.DefaultWorkingDirectory)/test-results'
So basically, I changed it from running only the tests in the test/Measter folder, to all the tests that are in /test.
Testing with Maester
The first test result (this is from the HTML file you get in artefacts):
For example, we can look into consent to applications. There are 3 tests that fail there, one for CISA, one for EIDSCA (I suspect there is one for CIS also, but it could be named something else) and one for Measter. When I click on one of the failed tests for this, I get a recommendation on how to fix this error. I did use the CISA recommendations to fix this and then run the test again. The result is improved:
There are now 2 more checks that have passed: the CISA and EIDSCA tests. But not the Maester test, and the reason is that it has a slightly different recommendation. The first two recommendations are to only allow Admins to give consent, but the Maester test has the recommendation to allow low-impact consent for users.
Conclusion
This is a great tool that can help organisations to improve their security in Entra and M365. It is also quite easy to create new custom tests, since it uses Pester and PowerShell.