Introduction to SAMATE
From SAMATE
The NIST Software Assurance Metrics And Tool Evaluation (SAMATE) project is dedicated to improving software assurance by developing methods to enable software tool evaluations, measuring the effectiveness of tools and techniques, and identifying gaps in tools and methods. The scope of the SAMATE project is broad: ranging from operating systems to firewalls, SCADA to web applications, source code security analyzers to correct-by-construction methods.
Contents |
Support Tool Evaluation
One of our goals is to establish a methodology for evaluating software assurance tools. We do this by developing tool specifications, test plans, and test sets. The results provide information for tool developers to improve tools, for users to make informed choices about acquiring and using software tools, and for interested parties to understand tool capabilities. Our efforts include:
The Software Assurance Reference Dataset (SARD) - A community repository of example code and other artifacts to help end users evaluate tools and developers test their methods. As of November 2016, the SARD consists of over 171 000 test cases which encompass a wide variety of flaws and languages.
Static Analysis Tools Exposition. The goals are to
- enable empirical research based on large test sets,
- encourage improvement of tools, and
- speed tool adoption by objectively demonstrating their use on real software.
Briefly, we pick a set of programs. Tool makers run their tools on them and return the tool reports. We perform a limited analysis of the reports and note interesting aspects. We and the participants report our experience and results at a workshop. We make the test set, some of the tool reports, and results publicly available later.
Source Code Security Analyzers – This class of software tools examines source code files for security weaknesses and potential vulnerabilities. We published a specification as NIST Special Publication 500-268 v1.1 and a draft test plan for source code security analyzers as NIST Special publication 500-270.
Web Vulnerability Scanners – These tools crawl a web application’s pages and search for vulnerabilities by simulating attacks on it. A specification is published as NIST Special Publication 500-269. A test framework for web application scanners appeared in a paper entitled “Building a Test Suite for Web Application Scanners” and published in 41st Hawaii International Conference on System Sciences (HICSS), January 2008.
An effort on Binary Code Scanners - Similar to source code security analyzers, this class of tool analyzes a compiled binary application, including libraries, and provides a report of code weakness over the entire application.
Studies on Software Assurance
Effect of Static Analysis tools on Software Security: Preliminary Investigation, Third Workshop on Quality of Protection (QoP), Oct 2007.
SAMATE and Evaluating Static Analysis Tools, International Conference on Reliable Software Technologies – Ada Europe, June 2007.
SAMATE Publications is a complete list of papers, workshops, and presentations.
Other Work
We started a new effort for capturing and sharing facts about a piece of software, including its claims. For example, a software fact sheet may have information about the product’s pedigree, development process, testing comprehension, security, safety, and quality.
History
SAMATE began in Fall, 2004.